Trimming search space for nearest neighbor determinations in point cloud compression

ABSTRACT

A search space for performing nearest neighbor searches for encoding point cloud data may be trimmed. Ranges of a space filling curve may be used to identify search space to exclude or reuse, instead of generating nearest neighbor search results for at least some of the points of a point cloud located within some of the ranges of the space filling curve. Additionally, neighboring voxels may be searched to identify any neighboring points missed during the trimmed search based on the ranges of the space filling curve.

PRIORITY CLAIM

This application is a continuation of U.S. patent application Ser. No.17/061,460, filed Oct. 1, 2020, which claims benefit of priority to U.S.Provisional Application Ser. No. 62/909,713, filed Oct. 2, 2019, andwhich also claims benefit of priority to U.S. Provisional ApplicationSer. No. 63/010,373, filed Apr. 15, 2020, all of which are incorporatedherein by reference in their entirety.

BACKGROUND Technical Field

This disclosure relates generally to compression and decompression ofpoint clouds comprising a plurality of points, each having associatedattribute information.

Description of the Related Art

Various types of sensors, such as light detection and ranging (LIDAR)systems, 3-D-cameras, 3-D scanners, etc. may capture data indicatingpositions of points in three dimensional space, for example positions inthe X, Y, and Z planes. Also, such systems may further capture attributeinformation in addition to spatial information for the respectivepoints, such as color information (e.g. RGB values), intensityattributes, reflectivity attributes, motion related attributes, modalityattributes, or various other attributes. In some circumstances,additional attributes may be assigned to the respective points, such asa time-stamp when the point was captured. Points captured by suchsensors may make up a “point cloud” comprising a set of points eachhaving associated spatial information and one or more associatedattributes. In some circumstances, a point cloud may include thousandsof points, hundreds of thousands of points, millions of points, or evenmore points. Also, in some circumstances, point clouds may be generated,for example in software, as opposed to being captured by one or moresensors. In either case, such point clouds may include large amounts ofdata and may be costly and time-consuming to store and transmit.

SUMMARY OF EMBODIMENTS

In some embodiments, a search space for performing nearest neighborsearches for encoding point cloud data may be trimmed. Ranges of a spacefilling curve may be used to identify a search space to exclude orreuse, instead of generating nearest neighbor search results for atleast some of the points of a point cloud located within some of theranges of the space filling curve.

In some embodiments, the space filling curve that is used is a Mortonorder, wherein Morton codes are determined for points of the point cloudfalling along the space filling curve. Also, in some embodiments, inaddition to using a trimmed search space resulting from a search onranges of the Morton codes on either side of a point being evaluated fornearest neighboring points, a Morton code of one or more neighboringvoxels that neighbor the point being evaluated is determined and asearch is done on the determined Morton codes for the points of thepoint cloud to see if the Morton code of the neighboring voxels includesa point of the point cloud. This may identify if nearest neighboringpoints included in neighboring voxels that have Morton codes outside ofthe trimmed search range.

Also, in some embodiments, Morton codes for neighboring voxels may bedetermined and searched for in an index of Morton codes for the pointcloud being compressed as an initial step. In such embodiments, if anumber of neighboring points found in the neighboring voxels is lessthan a desired number of nearest neighboring points to be used forprediction/level of detail generation, an additional search in a trimmedsearch range of Morton codes may be performed to identify additionalnearest neighboring points.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates a system comprising a sensor that capturesinformation for points of a point cloud and an encoder that compressesattribute information and/or spatial information of the point cloud,where the compressed point cloud information is sent to a decoder,according to some embodiments.

FIG. 1B illustrates a process for encoding attribute information of apoint cloud, according to some embodiments.

FIG. 1C illustrates representative views of point cloud information atdifferent stages of an encoding process, according to some embodiments.

FIG. 2A illustrates components of an encoder, according to someembodiments.

FIG. 2B illustrates components of a decoder, according to someembodiments.

FIG. 3 illustrates an example compressed attribute file, according tosome embodiments.

FIG. 4A illustrates a process for compressing attribute information of apoint cloud, according to some embodiments.

FIG. 4B illustrates predicting attribute values as part of compressingattribute information of a point cloud using adaptive distance basedprediction, according to some embodiments.

FIGS. 4C-4E illustrate parameters that may be determined or selected byan encoder and signaled with compressed attribute information for apoint cloud, according to some embodiments.

FIG. 5 illustrates a process for encoding attribute correction values,according to some embodiments.

FIGS. 6A-B illustrate an example process for compressing spatialinformation of a point cloud, according to some embodiments.

FIG. 7 illustrates another example process for compressing spatialinformation of a point cloud, according to some embodiments.

FIG. 8A illustrates an example process for decompressing compressedattribute information of a point cloud, according to some embodiments.

FIG. 8B illustrates predicting attribute values as part of decompressingattribute information of a point cloud using adaptive distance basedprediction, according to some embodiments.

FIG. 9 illustrates components an example encoder that generates ahierarchical level of detail (LOD) structure, according to someembodiments.

FIG. 10 illustrates an example process for determining points to beincluded at different refinement layers of a level of detail (LOD)structure, according to some embodiments.

FIG. 11A illustrates an example level of detail (LOD) structure,according to some embodiments.

FIG. 11B illustrates an example compressed point cloud file comprisinglevel of details for a point cloud (LODs), according to someembodiments.

FIG. 12A illustrates a method of encoding attribute information of apoint cloud, according to some embodiments.

FIG. 12B illustrates a method of decoding attribute information of apoint cloud, according to some embodiments.

FIG. 12C illustrates example neighborhood configurations of cubes of anoctree, according to some embodiments.

FIG. 12D illustrates an example look-ahead cube, according to someembodiments.

FIG. 12E illustrates, an example of 31 contexts that may be used toadaptively encode an index value of a symbol S using a binary arithmeticencoder, according to some embodiments.

FIG. 12F illustrates an example octree compression technique using abinary arithmetic encoder, cache, and look-ahead table, according tosome embodiments.

FIG. 13A illustrates a direct transformation that may be applied at anencoder to encode attribute information of a point could, according tosome embodiments.

FIG. 13B illustrates an inverse transformation that may be applied at adecoder to decode attribute information of a point cloud, according tosome embodiments.

FIG. 14 illustrates bounding box assignments to space filling curveranges for determining a minimum distance to points within the spacefilling curve ranges, according to some embodiments.

FIG. 15 illustrates a high-level flowchart for applying bounding shapesto trim search space for nearest neighbor searching, according to someembodiments.

FIG. 16 illustrates an example of nearest neighbor search result reuseaccording to ranges of a space filling curve, according to someembodiments.

FIG. 17 illustrates a high-level flowchart for applying bounding shapesto trim search space for nearest neighbor searching, according to someembodiments.

FIG. 18 illustrates points in discrete spaces that are used to improve anearest neighbor search, according to some embodiments.

FIG. 19 illustrates an example set of points and Morton codes for aspace filling curve, where the two points fall on the space fillingcurve, according to some embodiments.

FIG. 20 illustrates compressed point cloud information being used in a3-D telepresence application, according to some embodiments.

FIG. 21 illustrates compressed point cloud information being used in avirtual reality application, according to some embodiments.

FIG. 22 illustrates an example computer system that may implement anencoder or decoder, according to some embodiments.

This specification includes references to “one embodiment” or “anembodiment.” The appearances of the phrases “in one embodiment” or “inan embodiment” do not necessarily refer to the same embodiment.Particular features, structures, or characteristics may be combined inany suitable manner consistent with this disclosure.

“Comprising.” This term is open-ended. As used in the appended claims,this term does not foreclose additional structure or steps. Consider aclaim that recites: “An apparatus comprising one or more processor units. . . .” Such a claim does not foreclose the apparatus from includingadditional components (e.g., a network interface unit, graphicscircuitry, etc.).

“Configured To.” Various units, circuits, or other components may bedescribed or claimed as “configured to” perform a task or tasks. In suchcontexts, “configured to” is used to connote structure by indicatingthat the units/circuits/components include structure (e.g., circuitry)that performs those task or tasks during operation. As such, theunit/circuit/component can be said to be configured to perform the taskeven when the specified unit/circuit/component is not currentlyoperational (e.g., is not on). The units/circuits/components used withthe “configured to” language include hardware—for example, circuits,memory storing program instructions executable to implement theoperation, etc. Reciting that a unit/circuit/component is “configuredto” perform one or more tasks is expressly intended not to invoke 35U.S.C. § 112(f), for that unit/circuit/component. Additionally,“configured to” can include generic structure (e.g., generic circuitry)that is manipulated by software and/or firmware (e.g., an FPGA or ageneral-purpose processor executing software) to operate in manner thatis capable of performing the task(s) at issue. “Configure to” may alsoinclude adapting a manufacturing process (e.g., a semiconductorfabrication facility) to fabricate devices (e.g., integrated circuits)that are adapted to implement or perform one or more tasks.

“First,” “Second,” etc. As used herein, these terms are used as labelsfor nouns that they precede, and do not imply any type of ordering(e.g., spatial, temporal, logical, etc.). For example, a buffer circuitmay be described herein as performing write operations for “first” and“second” values. The terms “first” and “second” do not necessarily implythat the first value must be written before the second value.

“Based On.” As used herein, this term is used to describe one or morefactors that affect a determination. This term does not forecloseadditional factors that may affect a determination. That is, adetermination may be solely based on those factors or based, at least inpart, on those factors. Consider the phrase “determine A based on B.”While in this case, B is a factor that affects the determination of A,such a phrase does not foreclose the determination of A from also beingbased on C. In other instances, A may be determined based solely on B.

DETAILED DESCRIPTION

As data acquisition and display technologies have become more advanced,the ability to capture point clouds comprising thousands or millions ofpoints in 2-D or 3-D space, such as via LIDAR systems, has increased.Also, the development of advanced display technologies, such as virtualreality or augmented reality systems, has increased potential uses forpoint clouds. However, point cloud files are often very large and may becostly and time-consuming to store and transmit. For example,communication of point clouds over private or public networks, such asthe Internet, may require considerable amounts of time and/or networkresources, such that some uses of point cloud data, such as real-timeuses, may be limited. Also, storage requirements of point cloud filesmay consume a significant amount of storage capacity of devices storingthe point cloud files, which may also limit potential applications forusing point cloud data.

In some embodiments, an encoder may be used to generate a compressedpoint cloud to reduce costs and time associated with storing andtransmitting large point cloud files. In some embodiments, a system mayinclude an encoder that compresses attribute information and/or spatialinformation (also referred to herein as geometry information) of a pointcloud file such that the point cloud file may be stored and transmittedmore quickly than non-compressed point clouds and in a manner such thatthe point cloud file may occupy less storage space than non-compressedpoint clouds. In some embodiments, compression of spatial informationand/or attributes of points in a point cloud may enable a point cloud tobe communicated over a network in real-time or in near real-time. Forexample, a system may include a sensor that captures spatial informationand/or attribute information about points in an environment where thesensor is located, wherein the captured points and correspondingattributes make up a point cloud. The system may also include an encoderthat compresses the captured point cloud attribute information. Thecompressed attribute information of the point cloud may be sent over anetwork in real-time or near real-time to a decoder that decompressesthe compressed attribute information of the point cloud. Thedecompressed point cloud may be further processed, for example to make acontrol decision based on the surrounding environment at the location ofthe sensor. The control decision may then be communicated back to adevice at or near the location of the sensor, wherein the devicereceiving the control decision implements the control decision inreal-time or near real-time. In some embodiments, the decoder may beassociated with an augmented reality system and the decompressedattribute information may be displayed or otherwise used by theaugmented reality system. In some embodiments, compressed attributeinformation for a point cloud may be sent with compressed spatialinformation for points of the point cloud. In other embodiments, spatialinformation and attribute information may be separately encoded and/orseparately transmitted to a decoder.

In some embodiments, a system may include a decoder that receives one ormore point cloud files comprising compressed attribute information via anetwork from a remote server or other storage device that stores the oneor more point cloud files. For example, a 3-D display, a holographicdisplay, or a head-mounted display may be manipulated in real-time ornear real-time to show different portions of a virtual world representedby point clouds. In order to update the 3-D display, the holographicdisplay, or the head-mounted display, a system associated with thedecoder may request point cloud files from the remote server based onuser manipulations of the displays, and the point cloud files may betransmitted from the remote server to the decoder and decoded by thedecoder in real-time or near real-time. The displays may then be updatedwith updated point cloud data responsive to the user manipulations, suchas updated point attributes.

In some embodiments, a system, may include one or more LIDAR systems,3-D cameras, 3-D scanners, etc., and such sensor devices may capturespatial information, such as X, Y, and Z coordinates for points in aview of the sensor devices. In some embodiments, the spatial informationmay be relative to a local coordinate system or may be relative to aglobal coordinate system (for example, a Cartesian coordinate system mayhave a fixed reference point, such as a fixed point on the earth, or mayhave a non-fixed local reference point, such as a sensor location).

In some embodiments, such sensors may also capture attribute informationfor one or more points, such as color attributes, reflectivityattributes, velocity attributes, acceleration attributes, timeattributes, modalities, and/or various other attributes. In someembodiments, other sensors, in addition to LIDAR systems, 3-D cameras,3-D scanners, etc., may capture attribute information to be included ina point cloud. For example, in some embodiments, a gyroscope oraccelerometer, may capture motion information to be included in a pointcloud as an attribute associated with one or more points of the pointcloud. For example, a vehicle equipped with a LIDAR system, a 3-Dcamera, or a 3-D scanner may include the vehicle's direction and speedin a point cloud captured by the LIDAR system, the 3-D camera, or the3-D scanner. For example, when points in a view of the vehicle arecaptured they may be included in a point cloud, wherein the point cloudincludes the captured points and associated motion informationcorresponding to a state of the vehicle when the points were captured.

In some embodiments, attribute information may comprise string values,such as different modalities. For example attribute information mayinclude string values indicating a modality such as “walking”,“running”, “driving”, etc. In some embodiments, an encoder may comprisea “string-value” to integer index, wherein certain strings areassociated with certain corresponding integer values. In someembodiments, a point cloud may indicate a string value for a point byincluding an integer associated with the string value as an attribute ofthe point. The encoder and decoder may both store a common string valueto integer index, such that the decoder can determine string values forpoints based on looking up the integer value of the string attribute ofthe point in a string value to integer index of the decoder that matchesor is similar to the string value to integer index of the encoder.

In some embodiments, an encoder compresses and encodes spatialinformation of a point cloud to compress the spatial information inaddition to compressing attribute information for attributes of thepoints of the point cloud. For example, to compress spatial informationa K-D tree may be generated wherein, respective numbers of pointsincluded in each of the cells of the K-D tree are encoded. This sequenceof encoded point counts may encode spatial information for points of apoint cloud. Also, in some embodiments, a sub-sampling and predictionmethod may be used to compress and encode spatial information for apoint cloud. In some embodiments, the spatial information may bequantized prior to being compressed and encoded. Also, in someembodiments, compression of spatial information may be lossless. Thus, adecoder may be able to determine a same view of the spatial informationas an encoder. Also, an encoder may be able to determine a view of thespatial information a decoder will encounter once the compressed spatialinformation is decoded. Because, both an encoder and decoder may have orbe able to recreate the same spatial information for the point cloud,spatial relationships may be used to compress attribute information forthe point cloud.

For example, in many point clouds, attribute information betweenadjacent points or points that are located at relatively short distancesfrom each other may have high levels of correlation between attributes,and thus relatively small differences in point attribute values. Forexample, proximate points in a point cloud may have relatively smalldifferences in color, when considered relative to points in the pointcloud that are further apart.

In some embodiments, an encoder may include a predictor that determinesa predicted attribute value of an attribute of a point in a point cloudbased on attribute values for similar attributes of neighboring pointsin the point cloud and based on respective distances between the pointbeing evaluated and the neighboring points. In some embodiments,attribute values of attributes of neighboring points that are closer toa point being evaluated may be given a higher weighting than attributevalues of attributes of neighboring points that are further away fromthe point being evaluated. Also, the encoder may compare a predictedattribute value to an actual attribute value for an attribute of thepoint in the original point cloud prior to compression. A residualdifference, also referred to herein as an “attribute correction value”may be determined based on this comparison. An attribute correctionvalue may be encoded and included in compressed attribute informationfor the point cloud, wherein a decoder uses the encoded attributecorrection value to correct a predicted attribute value for the point,wherein the attribute value is predicted using a same or similarprediction methodology at the decoder that is the same or similar to theprediction methodology that was used at the encoder.

In some embodiments, to encode attribute values an encoder may generatean ordering of points of a point cloud based on spatial information forthe points of the point cloud. For example, the points may be orderedaccording a space-filling curve. In some embodiments, this ordering mayrepresent a Morton ordering of the points. The encoder may select afirst point as a starting point and may determine an evaluation orderfor other ones of the points of the point cloud based on minimumdistances from the starting point to a closest neighboring point, and asubsequent minimum distance from the neighboring point to the nextclosest neighboring point, etc. Also, in some embodiments, neighboringpoints may be determined from a group of points within a user-definedsearch range of an index value of a given point being evaluated, whereinthe index value and the search range values are values in an index ofthe points of the point cloud organized according to the space fillingcurve. In this way, an evaluation order for determining predictedattribute values of the points of the point cloud may be determined.Because the decoder may receive or re-create the same spatialinformation as the spatial information used by the encoder, the decodermay generate the same ordering of the points for the point cloud and maydetermine the same evaluation order for the points of the point cloud.

In some embodiments, an encoder may assign an attribute value for astarting point of a point cloud to be used to predict attribute valuesof other points of the point cloud. An encoder may predict an attributevalue for a neighboring point to the starting point based on theattribute value of the starting point and a distance between thestarting point and the neighboring point. The encoder may then determinea difference between the predicted attribute value for the neighboringpoint and the actual attribute value for the neighboring point includedin the non-compressed original point cloud. This difference may beencoded in a compressed attribute information file as an attributecorrection value for the neighboring point. The encoder may then repeata similar process for each point in the evaluation order. To predict theattribute value for subsequent points in the evaluation order, theencoder may identify the K-nearest neighboring points to a particularpoint being evaluated, wherein the identified K-nearest neighboringpoints have assigned or predicted attribute values. In some embodiments,“K” may be a configurable parameter that is communicated from an encoderto a decoder.

The encoder may determine a distance in X, Y, and Z space between apoint being evaluated and each of the identified neighboring points. Forexample, the encoder may determine respective Euclidian distances fromthe point being evaluated to each of the neighboring points. The encodermay then predict an attribute value for an attribute of the point beingevaluated based on the attribute values of the neighboring points,wherein the attribute values of the neighboring points are weightedaccording to an inverse of the distances from the point being evaluatedto the respective ones of the neighboring points. For example, attributevalues of neighboring points that are closer to the point beingevaluated may be given more weight than attribute values of neighboringpoints that are further away from the point being evaluated.

In a similar manner as described for the first neighboring point, theencoder may compare a predicted value for each of the other points ofthe point cloud to an actual attribute value in an originalnon-compressed point cloud, for example the captured point cloud. Thedifference may be encoded as an attribute correction value for anattribute of one of the other points that is being evaluated. In someembodiments, attribute correction values may be encoded in an order in acompressed attribute information file in accordance with the evaluationorder determined based on the space filling curve order. Because theencoder and the decoder may determine the same evaluation order based onthe spatial information for the point cloud, the decoder may determinewhich attribute correction value corresponds to which attribute of whichpoint based on the order in which the attribute correction values areencoded in the compressed attribute information file. Additionally, thestarting point and one or more attribute value(s) of the starting pointmay be explicitly encoded in a compressed attribute information filesuch that the decoder may determine the evaluation order starting withthe same point as was used to start the evaluation order at the encoder.Additionally, the one or more attribute value(s) of the starting pointmay provide a value of a neighboring point that a decoder uses todetermine a predicted attribute value for a point being evaluated thatis a neighboring point to the starting point.

In some embodiments, an encoder may determine a predicted value for anattribute of a point based on temporal considerations. For example, inaddition to or in place of determining a predicted value based onneighboring points in a same “frame” e.g. point in time as the pointbeing evaluated, the encoder may consider attribute values of the pointin adjacent and subsequent time frames.

FIG. 1A illustrates a system comprising a sensor that capturesinformation for points of a point cloud and an encoder that compressesattribute information of the point cloud, where the compressed attributeinformation is sent to a decoder, according to some embodiments.

System 100 includes sensor 102 and encoder 104. Sensor 102 captures apoint cloud 110 comprising points representing structure 106 in view 108of sensor 102. For example, in some embodiments, structure 106 may be amountain range, a building, a sign, an environment surrounding a street,or any other type of structure. In some embodiments, a captured pointcloud, such as captured point cloud 110, may include spatial andattribute information for the points included in the point cloud. Forexample, point A of captured point cloud 110 comprises X, Y, Zcoordinates and attributes 1, 2, and 3. In some embodiments, attributesof a point may include attributes such as R, G, B color values, avelocity at the point, an acceleration at the point, a reflectance ofthe structure at the point, a time stamp indicating when the point wascaptured, a string-value indicating a modality when the point wascaptured, for example “walking”, or other attributes. The captured pointcloud 110 may be provided to encoder 104, wherein encoder 104 generatesa compressed version of the point cloud (compressed attributeinformation 112) that is transmitted via network 114 to decoder 116. Insome embodiments, a compressed version of the point cloud, such ascompressed attribute information 112, may be included in a commoncompressed point cloud that also includes compressed spatial informationfor the points of the point cloud or, in some embodiments, compressedspatial information and compressed attribute information may becommunicated as separate files.

In some embodiments, encoder 104 may be integrated with sensor 102. Forexample, encoder 104 may be implemented in hardware or software includedin a sensor device, such as sensor 102. In other embodiments, encoder104 may be implemented on a separate computing device that is proximateto sensor 102.

FIG. 1B illustrates a process for encoding compressed attributeinformation of a point cloud, according to some embodiments. Also, FIG.1C illustrates representative views of point cloud information atdifferent stages of an encoding process, according to some embodiments.

At 152, an encoder, such as encoder 104, receives a captured point cloudor a generated point cloud. For example, in some embodiments a pointcloud may be captured via one or more sensors, such as sensor 102, ormay be generated in software, such as in a virtual reality or augmentedreality system. For example, 164 illustrates an example captured orgenerated point cloud. Each point in the point cloud shown in 164 mayhave one or more attributes associated with the point. Note that pointcloud 164 is shown in 2D for ease of illustration, but may includepoints in 3D space.

At 154, an ordering of the points of the point cloud is determinedaccording to a space filling curve. For example, a space filling curvemay fill a three dimensional space and points of a point cloud may beordered based on where they lie relative to the space filling curve. Forexample, a Morton code may be used to represent multi-dimensional datain one dimension, wherein a “Z-Order function” is applied to themultidimensional data to result in the one dimensional representation.In some embodiments, as discussed in more detail herein, the points mayalso be ordered into multiple levels of detail (LODs). In someembodiments, points to be included in respective levels of details(LODs) may be determined by ordering the points according to theirlocation along a space filling curve. For example, the points may beorganized according to their Morton codes.

In some embodiments, other space filling curves could be used. Forexample, techniques to map positions (e.g., in X, Y, Z coordinate form)to a space filling curve such as a Morton-order (or Z-order), Hillbertcurve, Peano curve, and so on may be used. In this way all of the pointsof the point cloud that are encoded and decoded using the spatialinformation may be organized into an index in the same order on theencoder and the decoder. In order to determine various refinementlevels, sampling rates, etc. the ordered index of the points may beused. For example, to divide a point cloud into four levels of detail,an index that maps a Morton value to a corresponding point may besampled, for example at a rate of four, where every fourth indexed pointis included in the lowest level refinement. For each additional level ofrefinement remaining points in the index that have not yet been sampledmay be sampled, for example every third index point, etc. until all ofthe points are sampled for a highest level of detail

At 156, an attribute value for one or more attributes of a startingpoint may be assigned to be encoded and included in compressed attributeinformation for the point cloud. As discussed above, predicted attributevalues for points of a point cloud may be determined based on attributevalues of neighboring points. However, an initial attribute value for atleast one point is provided to a decoder so that the decoder maydetermine attribute values for other points using at least the initialattribute value and attribute correction values for correcting predictedattribute values that are predicted based on the initial attributevalue. Thus, one or more attribute values for at least one startingpoint are explicitly encoded in a compressed attribute information file.Additionally, spatial information for the starting point may beexplicitly encoded such that the starting point may be identified by adecoder to determine which point of the points of the point cloud is tobe used as a starting point for generating an order according to aspace-filling curve. In some embodiments, a starting point may beindicated in other ways other than explicitly encoding the spatialinformation for the starting point, such as flagging the starting pointor other methods of point identification.

Because a decoder will receive an indication of a starting point andwill encounter the same or similar spatial information for the points ofthe point cloud as the encoder, the decoder may determine a same spacefilling curve order from the same starting point as was determined bythe encoder. Additionally, the decoder may determine a same processingorder as the encoder based on the space filling curve order determinedby the decoder.

At 158, for a current point being evaluated, a prediction/correctionevaluator of an encoder determines a predicted attribute value for anattribute of the point currently being evaluated. In some embodiments, apoint currently being evaluated may have more than one attribute.Accordingly, a prediction/correction evaluator of an encoder may predictmore than one attribute value for the point. For each point beingevaluated, the prediction/correction evaluator may identify a set ofnearest neighboring points that have assigned or predicted attributevalues. In some embodiments, a number of neighboring points to identify,“K”, may be a configurable parameter of an encoder and the encoder mayinclude configuration information in a compressed attribute informationfile indicating the parameter “K” such that a decoder may identify asame number of neighboring points when performing attribute prediction.The prediction/correction evaluator may then determine distances betweenthe point being evaluated and respective ones of the identifiedneighboring points. The prediction/correction evaluator may use aninverse distance interpolation method to predict an attribute value foreach attribute of the point being evaluated. The prediction/correctionevaluator may then predict an attribute value of the point beingevaluated based on an average of inverse-distance weighted attributevalues of the identified neighboring points.

For example, 166 illustrates a point (X,Y,Z) being evaluated whereinattribute A1 is being determined based on inverse distance weightedattribute values of eight identified neighboring points.

At 160, an attribute correction value is determined for each point. Theattribute correction value is determined based on comparing a predictedattribute value for each attribute of a point to corresponding attributevalues of the point in an original non-compressed point cloud, such asthe captured point cloud. For example, 168 illustrates an equation fordetermining attribute correction values, wherein a captured value issubtracted from a predicted value to determine an attribute correctionvalue. Note that while, FIG. 1B shows attribute values being predictedat 158 and attribute correction values being determined at 160, in someembodiments attribute correction values may be determined for a pointsubsequent to predicting an attribute value for the point. A next pointmay then be evaluated, wherein a predicted attribute value is determinedfor the point and an attribute correction value is determined for thepoint. Thus 158 and 160 may be repeated for each point being evaluated.In other embodiments, predicted values may be determined for multiplepoints and then attribute correction values may be determined. In someembodiments, predictions for subsequent points being evaluated may bebased on predicted attribute values or may be based on correctedattribute values or both. In some embodiments, both an encoder and adecoder may follow the same rules as to whether predicted values forsubsequent points are to be determined based on predicted or correctedattribute values.

At 162, the determined attribute correction values for the points of thepoint cloud, one or more assigned attribute values for the startingpoint, spatial information or other indicia of the starting point, andany configuration information to be included in a compressed attributeinformation file is encoded. As discussed in more detail in FIG. 5various encoding methods, such as arithmetic encoding and/or Golombencoding may be used to encode the attribute correction values, assignedattribute values, and the configuration information.

FIG. 2A illustrates components of an encoder, according to someembodiments.

Encoder 202 may be a similar encoder as encoder 104 illustrated in FIG.1A. Encoder 202 includes spatial encoder 204, space filling curve ordergenerator 210, prediction/correction evaluator 206, incoming datainterface 214, and outgoing data interface 208. Encoder 202 alsoincludes context store 216 and configuration store 218.

In some embodiments, a spatial encoder, such as spatial encoder 204, maycompress spatial information associated with points of a point cloud,such that the spatial information can be stored or transmitted in acompressed format. In some embodiments, a spatial encoder, may utilizeK-D trees to compress spatial information for points of a point cloud asdiscussed in more detail in regard to FIG. 7 . Also, in someembodiments, a spatial encoder, such as spatial encoder 204, may utilizea sub-sampling and prediction technique as discussed in more detail inregard to FIGS. 6A-B. In some embodiments, a spatial encoder, such asspatial encoder 204, may utilize Octrees to compress spatial informationfor points of a point cloud as discussed in more detail in regard toFIG. 12C-F.

In some embodiments, compressed spatial information may be stored ortransmitted with compressed attribute information or may be stored ortransmitted separately. In either case, a decoder receiving compressedattribute information for points of a point cloud may also receivecompressed spatial information for the points of the point cloud, or mayotherwise obtain the spatial information for the points of the pointcloud.

A space filling curve order generator, such as space filling curve ordergenerator 210, may utilize spatial information for points of a pointcloud to generate an indexed order of the points based on where thepoints fall along a space filling curve. For example Morton codes may begenerated for the points of the point cloud. Because a decoder isprovided or otherwise obtains the same spatial information for points ofa point cloud as are available at the encoder, a space filling curveorder determined by a space filling curve order generator of an encoder,such as space filling curve order generator 210 of encoder 202, may bethe same or similar as a space filling curve order generated by a spacefilling curve order generator of a decoder, such as space filling curveorder generator 228 of decoder 220.

A prediction/correction evaluator, such as prediction/correctionevaluator 206 of encoder 202, may determine predicted attribute valuesfor points of a point cloud based on an inverse distance interpolationmethod using attribute values of the K-nearest neighboring points of apoint for whom an attribute value is being predicted. Theprediction/correction evaluator may also compare a predicted attributevalue of a point being evaluated to an original attribute value of thepoint in a non-compressed point cloud to determine an attributecorrection value. In some embodiments, a prediction/correctionevaluator, such as prediction/correction evaluator 206 of encoder, 202may adaptively adjust a prediction strategy used to predict attributevalues of points in a given neighborhood of points based on ameasurement of the variability of the attribute values of the points inthe neighborhood.

An outgoing data encoder, such as outgoing data encoder 208 of encoder202, may encode attribute correction values and assigned attributevalues included in a compressed attribute information file for a pointcloud. In some embodiments, an outgoing data encoder, such as outgoingdata encoder 208, may select an encoding context for encoding a value,such as an assigned attribute value or an attribute correction value,based on a number of symbols included in the value. In some embodiments,values with more symbols may be encoded using an encoding contextcomprising Golomb exponential encoding, whereas values with fewersymbols may be encoded using arithmetic encoding. In some embodiments,encoding contexts may include more than one encoding technique. Forexample, a portion of a value may be encoded using arithmetic encodingwhile another portion of the value may be encoded using Golombexponential encoding. In some embodiments, an encoder, such as encoder202, may include a context store, such as context store 216, that storesencoding contexts used by an outgoing data encoder, such as outgoingdata encoder 208, to encode attribute correction values and assignedattribute values.

In some embodiments, an encoder, such as encoder 202, may also includean incoming data interface, such as incoming data interface 214. In someembodiments, an encoder may receive incoming data from one or moresensors that capture points of a point cloud or that capture attributeinformation to be associated with points of a point cloud. For example,in some embodiments, an encoder may receive data from an LIDAR system,3-D-camera, 3-D scanner, etc. and may also receive data from othersensors, such as a gyroscope, accelerometer, etc. Additionally, anencoder may receive other data such as a current time from a systemclock, etc. In some embodiments, such different types of data may bereceived by an encoder via an incoming data interface, such as incomingdata interface 214 of encoder 202.

In some embodiments, an encoder, such as encoder 202, may furtherinclude a configuration interface, such as configuration interface 212,wherein one or more parameters used by the encoder to compress a pointcloud may be adjusted via the configuration interface. In someembodiments, a configuration interface, such as configuration interface212, may be a programmatic interface, such as an API. Configurationsused by an encoder, such as encoder 202, may be stored in aconfiguration store, such as configuration store 218.

In some embodiments, an encoder, such as encoder 202, may include moreor fewer components than shown in FIG. 2A.

FIG. 2B illustrates components of a decoder, according to someembodiments.

Decoder 220 may be a similar decoder as decoder 116 illustrated in FIG.1A. Decoder 220 includes encoded data interface 226, spatial decoder222, space filling curve order generator 228, prediction evaluator 224,context store 232, configuration store 234, and decoded data interface220.

A decoder, such as decoder 220, may receive an encoded compressed pointcloud and/or an encoded compressed attribute information file for pointsof a point cloud. For example, a decoder, such as decoder 220, mayreceive a compressed attribute information file, such a compressedattribute information 112 illustrated in FIG. 1A or compressed attributeinformation file 300 illustrated in FIG. 3 . The compressed attributeinformation file may be received by a decoder via an encoded datainterface, such as encoded data interface 226. The encoded compressedpoint cloud may be used by the decoder to determine spatial informationfor points of the point cloud. For example, spatial information ofpoints of a point cloud included in a compressed point cloud may begenerated by a spatial information generator, such as spatialinformation generator 222. In some embodiments, a compressed point cloudmay be received via an encoded data interface, such as encoded datainterface 226, from a storage device or other intermediary source,wherein the compressed point cloud was previously encoded by an encoder,such as encoder 104.

In some embodiments, an encoded data interface, such as encoded datainterface 226, may decode spatial information. For example the spatialinformation may have been encoded using various encoding techniques suchas arithmetic encoding, Golomb encoding, etc. A spatial informationgenerator, such as spatial information generator 222, may receivedecoded spatial information from an encoded data interface, such asencoded data interface 226, and may use the decoded spatial informationto generate a representation of the geometry of the point cloud beingde-compressed. For example, decoded spatial information may be formattedas residual values to be used in a sub-sampled prediction method torecreate a geometry of a point cloud to be decompressed. In suchsituations, the spatial information generator 222, may recreate thegeometry of the point cloud being decompressed using decoded spatialinformation from encoded data interface 226, and space filling curveorder generator 228 may determine a space filling curve order for thepoint cloud being decompressed based on the recreated geometry for thepoint cloud being decompressed generated by spatial informationgenerator 222.

Once spatial information for a point cloud is determined and aspace-filling curve order has been determined, the space-filling curveorder may be used by a prediction evaluator of a decoder, such asprediction evaluator 224 of decoder 220, to determine an evaluationorder for determining attribute values of points of the point cloud.Additionally, the space-filling curve order may be used by a predictionevaluator, such as prediction evaluator 224, to identify nearestneighboring points to a point being evaluated.

A prediction evaluator of a decoder, such as prediction evaluator 224,may select a starting point such as based on an assigned starting pointincluded in a compressed attribute information file. In someembodiments, the compressed attribute information file may include oneor more assigned values for one or more corresponding attributes of thestarting point. In some embodiments, a prediction evaluator, such asprediction evaluator 224, may assign values to one or more attributes ofa starting point in a decompressed model of a point cloud beingdecompressed based on assigned values for the starting point included ina compressed attribute information file. A prediction evaluator, such asprediction evaluator 224, may further utilize the assigned values of theattributes of the starting point to determine attribute values ofneighboring points. For example, a prediction evaluator may select aneighboring point to the starting point as a next point to evaluate,wherein the neighboring point is selected based on an index order of thepoints according to the space-filling curve order. Note that because thespace-filling curve order is generated based on the same or similarspatial information at the decoder as was used to generate aspace-filling curve order at an encoder, the decoder may determine thesame evaluation order for evaluating the points of the point cloud beingdecompressed as was determined at the encoder by identifying nextnearest neighbors in an index according to the space-filling curveorder.

Once the prediction evaluator has identified the “K” nearest neighboringpoints to a point being evaluated, the prediction evaluator may predictone or more attribute values for one or more attributes of the pointbeing evaluated based on attribute values of corresponding attributes ofthe “K” nearest neighboring points. In some embodiments, an inversedistance interpolation technique may be used to predict an attributevalue of a point being evaluated based on attribute values ofneighboring points, wherein attribute values of neighboring points thatare at a closer distance to the point being evaluated are weighted moreheavily than attribute values of neighboring points that are at furtherdistances from the point being evaluated. In some embodiments, aprediction evaluator of a decoder, such as prediction evaluator 224 ofdecoder 220, may adaptively adjust a prediction strategy used to predictattribute values of points in a given neighborhood of points based on ameasurement of the variability of the attribute values of the points inthe neighborhood. For example, in embodiments wherein adaptiveprediction is used, the decoder may mirror prediction adaptationdecisions that were made at an encoder. In some embodiments, adaptiveprediction parameters may be included in compressed attributeinformation received by the decoder, wherein the parameters weresignaled by an encoder that generated the compressed attributeinformation. In some embodiments, a decoder may utilize one or moredefault parameters in the absence of a signaled parameter, or may inferparameters based on the received compressed attribute information.

A prediction evaluator, such as prediction evaluator 224, may apply anattribute correction value to a predicted attribute value to determinean attribute value to include for the point in a decompressed pointcloud. In some embodiments, an attribute correction value for anattribute of a point may be included in a compressed attributeinformation file. In some embodiments, attribute correction values maybe encoded using one of a plurality of supported coding contexts,wherein different coding contexts are selected to encode differentattribute correction values based on a number of symbols included in theattribute correction value. In some embodiments, a decoder, such asdecoder 220, may include a context store, such as context store 232,wherein the context store stores a plurality of encoding context thatmay be used to decode assigned attribute values or attribute correctionvalues that have been encoded using corresponding encoding contexts atan encoder.

A decoder, such as decoder 220, may provide a decompressed point cloudgenerated based on a received compressed point cloud and/or a receivedcompressed attribute information file to a receiving device orapplication via a decoded data interface, such as decoded data interface230. The decompressed point cloud may include the points of the pointcloud and attribute values for attributes of the points of the pointcloud. In some embodiments, a decoder may decode some attribute valuesfor attributes of a point cloud without decoding other attribute valuesfor other attributes of a point cloud. For example, a point cloud mayinclude color attributes for points of the point cloud and may alsoinclude other attributes for the points of the point cloud, such asvelocity, for example. In such a situation, a decoder may decode one ormore attributes of the points of the point cloud, such as the velocityattribute, without decoding other attributes of the points of the pointcloud, such as the color attributes.

In some embodiments, the decompressed point cloud and/or decompressedattribute information file may be used to generate a visual display,such as for a head mounted display. Also, in some embodiments, thedecompressed point cloud and/or decompressed attribute information filemay be provided to a decision making engine that uses the decompressedpoint cloud and/or decompressed attribute information file to make oneor more control decisions. In some embodiments, the decompressed pointcloud and/or decompressed attribute information file may be used invarious other applications or for various other purposes.

FIG. 3 illustrates an example compressed attribute information file,according to some embodiments. Attribute information file 300 includesconfiguration information 302, point cloud data 304, and point attributecorrection values 306. In some embodiments, point cloud file 300 may becommunicated in parts via multiple packets. In some embodiments, not allof the sections shown in attribute information file 300 may be includedin each packet transmitting compressed attribute information. In someembodiments, an attribute information file, such as attributeinformation file 300, may be stored in a storage device, such as aserver that implements an encoder or decoder, or other computing device.In some embodiments, additional configuration information may includeadaptive prediction parameters, such as a variability measurementtechnique to use to determine a variability measurement for aneighborhood of points, a threshold variability value to trigger use ofa particular prediction procedure, one or more parameters fordetermining a size of a neighborhood of points for which variability isto be determined, etc.

FIG. 4A illustrates a process for compressing attribute information of apoint cloud, according to some embodiments.

At 402, an encoder receives a point cloud that includes attributeinformation for at least some of the points of the point cloud. Thepoint cloud may be received from one or more sensors that capture thepoint cloud, or the point cloud may be generated in software. Forexample, a virtual reality or augmented reality system may havegenerated the point cloud.

At 404, the spatial information of the point cloud, for example X, Y,and Z coordinates for the points of the point cloud may be quantized. Insome embodiments, coordinates may be rounded off to the nearestmeasurement unit, such as a meter, centimeter, millimeter, etc.

At 406, the quantized spatial information is compressed. In someembodiments, spatial information may be compressed using a sub-samplingand subdivision prediction technique as discussed in more detail inregard to FIGS. 6A-B. Also, in some embodiments, spatial information maybe compressed using a K-D tree compression technique as discussed inmore detail in regard to FIG. 7 , or may be compressed using an Octreecompression technique. In some embodiments, other suitable compressiontechniques may be used to compress spatial information of a point cloud.

At 408, the compressed spatial information for the point cloud isencoded as a compressed point cloud file or a portion of a compressedpoint cloud file. In some embodiments, compressed spatial informationand compressed attribute information may be included in a commoncompressed point cloud file, or may be communicated or stored asseparate files.

At 412, the received spatial information of the point cloud is used togenerate an indexed point order according to a space-filling curve. Insome embodiments, the spatial information of the point cloud may bequantized before generating the order according to the space-fillingcurve. Additionally, in some embodiments wherein a lossy compressiontechnique is used to compress the spatial information of the pointcloud, the spatial information may be lossy encoded and lossy decodedprior to generating the order according to the space filling curve. Inembodiments that utilize lossy compression for spatial information,encoding and decoding the spatial information at the encoder may ensurethat an order according to a space filling curve generated at theencoder will match an order according to the space filling curve thatwill be generated at a decoder using decoded spatial information thatwas previously lossy encoded.

Additionally, in some embodiments, at 410, attribute information forpoints of the point cloud may be quantized. For example attribute valuesmay be rounded to whole numbers or to particular measurement increments.In some embodiments wherein attribute values are integers, such as whenintegers are used to communicate string values, such as “walking”,“running”, “driving”, etc., quantization at 410 may be omitted.

At 414, attribute values for a starting point are assigned. The assignedattribute values for the starting point are encoded in a compressedattribute information file along with attribute correction values.Because a decoder predicts attribute values based on distances toneighboring points and attribute values of neighboring points, at leastone attribute value for at least one point is explicitly encoded in acompressed attribute file. In some embodiments, points of a point cloudmay comprise multiple attributes and at least one attribute value foreach type of attribute may be encoded for at least one point of thepoint cloud, in such embodiments. In some embodiments, a starting pointmay be a first point evaluated when determining the order according tothe space filling curve at 412. In some embodiments, an encoder mayencode data indicating spatial information for a starting point and/orother indicia of which point of the point cloud is the starting point orstarting points. Additionally, the encoder may encode attribute valuesfor one or more attributes of the starting point.

At 416, the encoder determines an evaluation order for predictingattribute values for other points of the point cloud, other than thestarting point, said predicting and determining attribute correctionvalues, may be referred to herein as “evaluating” attributes of a point.The evaluation order may be determined based on the order according tothe space filling curve.

At 418, a neighboring point of the starting point or of a subsequentpoint being evaluated is selected. In some embodiments, a neighboringpoint to be next evaluated may be selected based on the neighboringpoint being a next point in an indexed order of points according to aspace filling curve.

At 420, the “K” nearest neighboring points to the point currently beingevaluated are determined. The parameter “K” may be a configurableparameter selected by an encoder or provided to an encoder as a userconfigurable parameter. In order to select the “K” nearest neighboringpoints, an encoder may identify the first “K” nearest points to a pointbeing evaluated according to the indexed order of points determined at412 and respective distances between the points. For example, instead ofdetermining the absolute nearest neighboring points to a point beingevaluated, an encoder may select a group of points of the point cloudhaving index values in the index according to the space-filling curvethat are within a user defined search range, e.g. 8, 16, 32, 64, etc. ofan index value of a particular point being evaluated. The encoder maythen utilize distances within the group of points to select the “K”nearest neighboring points to use for prediction. In some embodiments,only points having assigned attribute values or for which predictedattribute values have already been determined may be included in the “K”nearest neighboring points. In some embodiments various numbers ofpoints may be identified. For example, in some embodiments, “K” may be 5points, 10 points, 16 points, etc. Because a point cloud comprisespoints in 3-D space a particular point may have multiple neighboringpoints in multiple planes. In some embodiments, an encoder and a decodermay be configured to identify points as the “K” nearest neighboringpoints regardless of whether or not a value has already been predictedfor the point. Also, in some embodiments, attribute values for pointsused in predication may be previously predicted attribute values orcorrected predicted attribute values that have been corrected based onapplying an attribute correction value. In either case, an encoder and adecoder may be configured to apply the same rules when identifying the“K” nearest neighboring points and when predicting an attribute value ofa point based on attribute values of the “K” nearest neighboring points.

At 422, one or more attribute values are determined for each attributeof the point currently being evaluated. The attribute values may bedetermined based on an inverse distance interpolation. The inversedistance interpolation may interpolate the predicted attribute valuebased on the attribute values of the “K” nearest neighboring points. Theattribute values of the “K” nearest neighboring points may be weightedbased on respective distances between respective ones of the “K” nearestneighboring points and the point being evaluated. Attribute values ofneighboring points that are at shorter distances from the pointcurrently being evaluated may be weighted more heavily than attributevalues of neighboring points that are at greater distances from thepoint currently being evaluated.

At 424, attribute correction values are determined for the one or morepredicted attribute values for the point currently being evaluated. Theattribute correction values may be determined based on comparing thepredicted attribute values to corresponding attribute values for thesame point (or a similar point) in the point cloud prior to attributeinformation compression. In some embodiments, quantized attributeinformation, such as the quantized attribute information generated at410, may be used to determine attribute correction values. In someembodiments, an attribute correction value may also be referred to as a“residual error” wherein the residual error indicates a differencebetween a predicted attribute value and an actual attribute value.

At 426, it is determined if there are additional points in the pointcloud for which attribute correction values are to be determined. Ifthere are additional points to evaluate, the process reverts to 418 andthe next point in the evaluation order is selected to be evaluated. Theprocess may repeat steps 418-426 until all or a portion of all of thepoints of the point cloud have been evaluated to determine predictedattribute values and attribute correction values for the predictedattribute values.

At 428, the determined attribute correction values, the assignedattribute values, and any configuration information for decoding thecompressed attribute information file, such as a parameter “K”, isencoded.

Adaptive Attribute Prediction

In some embodiments, an encoder as described above may furtheradaptively change a prediction strategy and/or a number of points usedin a given prediction strategy based on attribute values of neighboringpoints. Also, a decoder may similarly adaptively change a predictionstrategy and/or a number of points used in a given prediction strategybased on reconstructed attribute values of neighboring points.

For example, a point cloud may include points representing a road wherethe road is black with a white stripe on the road. A default nearestneighbor prediction strategy may be adaptively changed to take intoaccount the variability of attribute values for points representing thewhite line and the black road. Because these points have a largedifference in attribute values, a default nearest neighbor predictionstrategy may result in blurring of the white line and/or high residualvalues that decrease a compression efficiency. However, an updatedprediction strategy may account for this variability by selecting abetter suited prediction strategy and/or by using less points in aK-nearest neighbor prediction. For example, for the black road, notusing the white line points in a K-nearest neighbor prediction.

In some embodiments, before predicting an attribute value for a point P,an encoder or decoder may compute the variability of attribute values ofpoints in a neighborhood of point P, for example the K-nearestneighboring points. In some embodiments, variability may be computedbased on a variance, a maximum difference between any two attributevalues (or reconstructed attribute values) of the points neighboringpoint P. In some embodiments, variability may be computed based on aweighted average of the neighboring points, wherein the weighted averageaccounts for distances of the neighboring points to point P. In someembodiments, variability for a group of neighboring points may becomputed based on a weighted averages for attributes for the neighboringpoints and taking into account distances to the neighboring points. Forexample,Variability=E[(X−weighted mean(X))²]In the above equation, E is the mean attribute value of the points inthe neighborhood of point P, the weighted mean(X) is a weighted mean ofthe attribute values of the points in the neighborhood of point P thattakes into account the distances of the neighboring points from point P.In some embodiments, the variability may be calculated as the maximumdifference compared to the mean value of the attributes, E(X), theweighted mean of the attributes, weighted mean(X), or the median valueof the attributes, median(X). In some embodiments, the variability maybe calculated using the average of the values corresponding to the xpercent, e.g. x=10 that have the largest difference as compared to themean value of the attributes, E(X), the weighted mean of the attributes,weighted mean(X), or the median value of the attributes, median(X).

In some embodiments, if the calculated variability of the attributes ofthe points in the neighborhood of point P is greater than a thresholdvalue, then a rate-distortion optimization may be applied. For example,a rate-distortion optimization may reduce a number of neighboring pointsused in a prediction or switch to a different prediction technique. Insome embodiments, the threshold may be explicitly written in thebit-stream. Also, in some embodiments, the threshold may be adaptivelyadjusted per point cloud, or sub-block of the point cloud or for anumber of points to be encoded. For example, a threshold may be includedin compressed attribute information file 350 as additional configurationinformation included in configuration information 302, as described inFIG. 3 , or may be included in compressed attribute file 1150 asadditional configuration information included in configurationinformation 1152, as described below in regard to FIG. 11B.

In some embodiments, different distortion measures may be used in arate-distortion optimization procedure, such as sum of squares error,weighted sum of squares error, sum of absolute differences, or weightedsum of absolute differences.

In some embodiments, distortion could be computed independently for eachattribute, or multiple attributes corresponding to the same sample andcould be considered, and appropriately weighted. For example, distortionvalues for R, G, B or Y, U, V could be computed and then combinedtogether linearly or non-linearly to generate an overall distortionvalue.

In some embodiments, advanced techniques for rate distortionquantization, such as trellis based quantization could also beconsidered where, instead of considering a single point in isolationmultiple points are coded jointly. The coding process, for example, mayselect to encode all these multiple points using the method that resultsin minimizing a cost function of the form J=D+lambda*Rate, where D isthe overall distortion for all these points, and Rate is the overallrate cost for coding these points.

In some embodiments, an encoder, such as encoder 202, may explicitlyencode an index value of a chosen prediction strategy for a point cloud,for a level of detail of a point cloud, or for a group of points withina level of detail of a point cloud, wherein the decoder has access to aninstance of the index and can determine the chosen prediction strategybased on the received index value. The decoder may apply the chosenprediction strategy for the set of points for which the rate-distortionoptimization procedure is being applied. In some embodiments, there maybe a default prediction strategy and the decoder may apply the defaultprediction strategy if no rate-distortion optimization procedure isspecified in the encoded bit stream. Also, in some embodiments a defaultprediction strategy may be applied if no variability threshold is met.

For example, FIG. 4B illustrates predicting attribute values as part ofcompressing attribute information of a point cloud using adaptivedistance based prediction, according to some embodiments.

In some embodiments in which adaptive distance based prediction isemployed, predicting attribute values as described in elements 420 and422 of FIG. 4A may further include steps such as 450-456 to select aprediction procedure to be used to predict the attribute values for thepoints. In some embodiments the selected prediction procedure may be aK-nearest neighbor prediction procedure, as described herein and inregard to element 420 in FIG. 4A. In some embodiments, the selectedprediction procedure may be a modified K-nearest neighbor predictionprocedure, wherein fewer points are included in the number of nearestneighbors used to perform the adaptive prediction than a number ofpoints used to predict attribute values for portions of the point cloudwith less variability. In some embodiments, the selected predictionprocedure may be that the point for which an attribute value is beingpredicted simply uses the attribute value of the nearest point to thepoint for which the attribute value is being predicted, if thevariability of the neighboring points exceeds a threshold associatedwith this prediction procedure. In some embodiments, other predictionprocedures may be used depending on the variability of points in aneighborhood of a point for which an attribute value is being predicted.For example, in some embodiments, other prediction procedures, such as anon-distance based interpolation procedure may be used, such asbarycentric interpolation, natural neighbor interpolation, moving leastsquares interpolation, or other suitable interpolation techniques.

At 450, the encoder identifies a set of neighboring points for aneighborhood of a point of the point cloud for which an attribute valueis being predicted. In some embodiments, the set of neighboring pointsof the neighborhood may be identified using a K-nearest neighbortechnique as described herein. In some embodiments, points to be used todetermine variability may be identified in other manners. For example,in some embodiments, a neighborhood of points used for variabilityanalysis may be defined to include more or fewer points or points withina greater or smaller distance from the given point than are used topredict attribute values based on inverse distance based interpolationusing the K-nearest neighboring points. In some embodiments, whereinparameters used to identify the neighborhood points for determiningvariability differ from the parameters used in a K-nearest neighborprediction, the differing parameters or data from which the differingparameter may be determined is signaled in a bit stream encoded by theencoder.

At 452, the variability of the attribute values of the neighboringpoints is determined. In some embodiment, each attribute valuevariability may be determined separately. For example, for points withR, G, B attribute values each attribute value (e.g. each of R, G, and B)may have their respective variabilities determined separately. Also, insome embodiments trellis quantization may be used wherein a set ofattributes such as RGB that have correlated values may be determined asa common variability. For example, in the example discussed above withregard to the white stripe on the black road, the large variability in Rmay also apply to B and G, thus it is not necessary to determinevariability for each of R, G, and B separately. Instead the relatedattribute values can be considered as a group and a common variabilityfor the correlated attributes can be determined.

In some embodiments, the variability of the attributes in theneighborhood of point P may be determined using: a sum of square errorsvariability technique, a distance weighted sum of square errorsvariability technique, a sum of absolute differences variabilitytechnique, a distance weighted sum of absolute differences variabilitytechnique, or other suitable variability technique. In some embodimentsthe encoder may select a variability technique to be used for a givenpoint P, and may encode in a bit stream encoded by the encoder an indexvalue for an index of variability techniques, wherein the decoderincludes the same index and can determine which variability technique touse for point P based on the encoded index value.

At 454 through 456 it is determined whether or not the variabilitydetermined at 452 exceeds one or more variability thresholds. If so, acorresponding prediction technique that corresponds with the exceededvariability threshold is used to predict the attribute value or valuesfor the point P. In some embodiments, multiple prediction procedures maybe supported. For example, element 458 indicates using a firstprediction procedure if a first variability threshold is exceeded andelement 460 indicates using another prediction procedure if anothervariability threshold is exceeded. Furthermore, 462 indicates using adefault prediction procedure, such as a non-modified K-nearest neighborprediction procedure if the variability thresholds 1 through N are notexceeded. In some embodiments, a single variability threshold and asingle alternate prediction procedure may be used in addition to adefault prediction procedure. In some embodiments, any number of “N”variability thresholds and corresponding prediction procedures may beused.

For example, in some embodiments, if a first variability threshold isexceeded a first prediction procedure may be to use fewer neighboringpoints than are used in the default K-nearest neighbor predictionprocedure. Also, if a second variability threshold is exceeded, a secondprediction procedure may be to use only the nearest point to determinethe attribute value of the point P. Thus, in such embodiments, mediumvariability may cause some outlier points to be omitted under the firstprediction procedure and higher variability may cause all but theclosest neighboring point to be omitted from the prediction procedure,while if variability is low, the K-nearest neighboring points are usedin the default prediction procedure.

FIGS. 4C-4E illustrate parameters that may be determined or selected byan encoder and signaled with compressed attribute information for apoint cloud, according to some embodiments.

In FIG. 4C at 470, an encoder may select a variability measurementtechnique to be used to determine attribute variability for points in aneighborhood of a point P for which an attribute value is beingpredicted. In some embodiments, the encoder may utilize a ratedistortion optimization framework to determine which variabilitymeasurement technique to use. At 472 the encoder may include, in a bitstream encoded by the encoder, a signal indicating which variabilitytechnique was selected.

In FIG. 4D at 480, an encoder may determine a variability threshold forpoints in a neighborhood of a point P for which an attribute value isbeing predicted. In some embodiments, the encoder may utilize a ratedistortion optimization framework to determine the variabilitythreshold. At 482 the encoder may include in a bit stream, encoded bythe encoder, a signal indicating which variability threshold was used bythe encoder to perform prediction.

In FIG. 4E at 490, an encoder may determine or select a neighborhoodsize for use in determining variability. For example, the encoder mayuse a rate distortion optimization technique to determine how big orsmall of a neighborhood of points to use in determining variability forpoint P. At 492, the encoder may include in a bit stream, encoded by theencoder, one or more values for defining the neighborhood size. Forexample, the encoder may signal a minimum distance from point P, amaximum distance from point P, a total number of neighboring points toinclude, etc. and these parameters may define which points are includedin the neighborhood points for point P that are considered indetermining variability.

In some embodiments, one or more of the variability technique,variability threshold, or neighborhood size may not be signaled and mayinstead be determined at a decoder using a predetermined parameter knownto both the encoder and decoder. In some embodiments, a decoder mayinfer one or more of the variability technique, variability threshold,or neighborhood size to be used based on other data, such as spatialinformation for the point cloud.

Once the attribute values are predicted using the appropriatecorresponding prediction procedure at 858-862, the decoder may proceedto 820 and apply attribute correction values received in the encoded bitstream to adjust the predicted attribute values. In some embodiments,using adaptive prediction as described herein at the encoder and decodermay reduce a number of bits necessary to encode the attribute correctionvalues and may also reduce distortion of a re-constructed point cloudre-constructed at the decoder using the prediction procedures and thesignaled attribute correction values.

Example Process for Encoding Attribute Values and/or AttributeCorrection Values

The attribute correction values, the assigned attribute values, and anyconfiguration information may be encoded using various encodingtechniques.

For example, FIG. 5 illustrates a process for encoding attributecorrection values, according to some embodiments. At 502, an attributecorrection value for a point whose values (e.g. attribute correctionvalues) are being encoded is converted to an unsigned value. Forexample, in some embodiments, attribute correction values that arenegative values may be assigned odd numbers and attribute correctionvalues that are positive values may be assigned even numbers. Thus,whether or not the attribute correction value is positive or negativemay be implied based on whether or not a value of the attributecorrection value is an even number or an odd number. In someembodiments, assigned attribute values may also be converted intounsigned values. In some embodiments, attribute values may all bepositive values, for example in the case of integers that are assignedto represent string values, such as “walking”, “running”, “driving” etc.In such cases, 502 may be omitted.

At 504, an encoding context is selected for encoding a first value for apoint. The value may be an assigned attribute value or may be anattribute correction value, for example. The encoding context may beselected from a plurality of supported encoding contexts. For example, acontext store, such as context store 216 of an encoder, such as encoder202, as illustrated in FIG. 2A, may store a plurality of supportedencoding context for encoding attribute values or attribute correctionvalues for points of a point cloud. In some embodiments, an encodingcontext may be selected based on characteristics of a value to beencoded. For example, some encoding contexts may be optimized forencoding values with certain characteristics while other encodingcontexts may be optimized for encoding values with othercharacteristics.

In some embodiments, an encoding context may be selected based on aquantity or variety of symbols included in a value to be encoded. Forexample, values with fewer or less diverse symbols may be encoded usingarithmetic encoding techniques, while values with more symbols or morediverse symbols may be encoding using exponential Golomb encodingtechniques. In some embodiments, an encoding context may encode portionsof a value using more than one encoding technique. For example, in someembodiments, an encoding context may indicate that a portion of a valueis to be encoded using an arithmetic encoding technique and anotherportion of the value is to be encoded using a Golomb encoding technique.In some embodiments, an encoding context may indicate that a portion ofa value below a threshold is to be encoded using a first encodingtechnique, such as arithmetic encoding, whereas another portion of thevalue exceeding the threshold is to be encoded using another encodingtechnique, such as exponential Golomb encoding. In some embodiments, acontext store may store multiple encoding contexts, wherein eachencoding context is suited for values having particular characteristics.

At 506, a first value (or additional value) for the point may be encodedusing the encoding context selected at 504. At 508 it is determined ifthere are additional values for the point that are to be encoded. Ifthere are additional values for the point to be encoded, the additionalvalues may be encoded, at 506, using the same selected encodingtechnique that was selected at 504. For example, a point may have a“Red”, a “Green”, and a “Blue” color attribute. Because differencesbetween adjacent points in the R, G, B color space may be similar,attribute correction values for the Red attribute, Green attribute, andBlue attribute may be similar. Thus, in some embodiments, an encoder mayselect an encoding context for encoding attribute correction values fora first one of the color attributes, for example the Red attribute, andmay use the same encoding context for encoding attribute correctionvalues for the other color attributes, such as the Green attribute andthe Blue attribute.

At 510 encoded values, such as encoded assigned attribute values andencoded attribute correction values may be included in a compressedattribute information file. In some embodiments, the encoded values maybe included in the compressed attribute information file in accordancewith the evaluation order determined for the point cloud based on aspace filling curve order. Thus a decoder may be able to determine whichencoded value goes with which attribute of which point based on theorder in which encoded values are included in a compressed attributeinformation file. Additionally, in some embodiments, data may beincluded in a compressed attribute information file indicatingrespective ones of the encoding contexts that were selected to encoderespective ones of the values for the points.

Example Processes for Encoding Spatial Information

FIGS. 6A-B illustrate an example process for compressing spatialinformation of a point cloud, according to some embodiments.

At 602, an encoder receives a point cloud. The point cloud may be acaptured point cloud from one or more sensors or may be a generatedpoint cloud, such as a point cloud generated by a graphics application.For example, 604 illustrates points of an un-compressed point cloud.

At 606, the encoder sub-samples the received point cloud to generate asub-sampled point cloud. The sub-sampled point cloud may include fewerpoints than the received point cloud. For example, the received pointcloud may include hundreds of points, thousands of points, or millionsof points and the sub-sampled point cloud may include tens of points,hundreds of points or thousands of points. For example, 608 illustratessub-sampled points of a point cloud received at 602, for example asub-sampling of the points of the point cloud in 604.

In some embodiments, the encoder may encode and decode the sub-sampledpoint cloud to generate a representative sub-sampled point cloud thedecoder will encounter when decoding the compressed point cloud. In someembodiments, the encoder and decoder may execute a lossycompression/decompression algorithm to generate the representativesub-sampled point cloud. In some embodiments, spatial information forpoints of a sub-sampled point cloud may be quantized as part ofgenerating a representative sub-sampled point cloud. In someembodiments, an encoder may utilize lossless compression techniques andencoding and decoding the sub-sampled point cloud may be omitted. Forexample, when using lossless compression techniques the originalsub-sampled point cloud may be representative of a sub-sampled pointcloud the decoder will encounter because in lossless compression datamay not be lost during compression and decompression.

At 610, the encoder identifies subdivision locations between points ofthe sub-sampled point cloud according to configuration parametersselected for compression of the point cloud or according to fixedconfiguration parameters. The configuration parameters used by theencoder that are not fixed configuration parameters are communicated toan encoder by including values for the configuration parameters in acompressed point cloud. Thus, a decoder may determine the samesubdivision locations as the encoder evaluated based on subdivisionconfiguration parameters included in the compressed point cloud. Forexample, 612 illustrates identified sub-division locations betweenneighboring points of a sub-sampled point cloud.

At 614, the encoder determines for respective ones of the subdivisionlocations whether a point is to be included or not included at thesubdivision location in a decompressed point cloud. Data indicating thisdetermination is encoded in the compressed point cloud. In someembodiments, the data indicating this determination may be a single bitthat if “true” means a point is to be included and if “false” means apoint is not to be included. Additionally, an encoder may determine thata point that is to be included in a decompressed point cloud is to berelocated relative to the subdivision location in the decompressed pointcloud. For example 616, shows some points that are to be relocatedrelative to a subdivision location. For such points, the encoder mayfurther encode data indicating how to relocate the point relative to thesubdivision location. In some embodiments, location correctioninformation may be quantized and entropy encoded. In some embodiments,the location correction information may comprise delta X, delta Y,and/or delta Z values indicating how the point is to be relocatedrelative to the subdivision location. In other embodiments, the locationcorrection information may comprise a single scalar value whichcorresponds to the normal component of the location correctioninformation computed as follows:ΔN=([X _(A) ,Y _(A) ,Z _(A)]−[X,Y,Z])·[Normal Vector]

In the above equation, delta N is a scalar value indicating locationcorrection information that is the difference between the relocated oradjusted point location relative to the subdivision location (e.g.[X_(A), Y_(A), Z_(A)]) and the original subdivision location (e.g. [X,Y, Z]). The cross product of this vector difference and the normalvector at the subdivision location results in the scalar value delta N.Because a decoder can determine, the normal vector at the subdivisionlocation, and can determine the coordinates of the subdivision location,e.g. [X, Y, Z], the decoder can also determine the coordinates of theadjusted location, e.g. [X_(A), Y_(A), Z_(A)], by solving the aboveequation for the adjusted location, which represents a relocatedlocation for a point relative to the subdivision location. In someembodiments, the location correction information may be furtherdecomposed into a normal component and one or more additional tangentialcomponents. In such an embodiment, the normal component, e.g. delta N,and the tangential component(s) may be quantized and encoded forinclusion in a compressed point cloud.

In some embodiments, an encoder may determine whether one or moreadditional points (in addition to points included at subdivisionlocations or points included at locations relocated relative tosubdivision locations) are to be included in a decompressed point cloud.For example, if the original point cloud has an irregular surface orshape such that subdivision locations between points in the sub-sampledpoint cloud do not adequately represent the irregular surface or shape,the encoder may determine to include one or more additional points inaddition to points determined to be included at subdivision locations orrelocated relative to subdivision locations in the decompressed pointcloud. Additionally, an encoder may determine whether one or moreadditional points are to be included in a decompressed point cloud basedon system constraints, such as a target bitrate, a target compressionratio, a quality target metric, etc. In some embodiments, a bit budgetmay change due to changing conditions such as network conditions,processor load, etc. In such embodiments, an encoder may adjust aquantity of additional points that are encoded to be included in adecompressed point cloud based on a changing bit budget. In someembodiments, an encoder may include additional points such that a bitbudget is consumed without being exceeded. For example, when a bitbudget is higher, an encoder may include more additional points toconsume the bit budget (and enhance quality) and when the bit budget isless, the encoder may include fewer additional points such that the bitbudget is consumed but not exceeded.

In some embodiments, an encoder may further determine whether additionalsubdivision iterations are to be performed. If so, the points determinedto be included, relocated, or additionally included in a decompressedpoint cloud are taken into account and the process reverts to 610 toidentify new subdivision locations of an updated sub-sampled point cloudthat includes the points determined to be included, relocated, oradditionally included in the decompressed point cloud. In someembodiments, a number of subdivision iterations to be performed (N) maybe a fixed or configurable parameter of an encoder. In some embodiments,different subdivision iteration values may be assigned to differentportions of a point cloud. For example, an encoder may take into accounta point of view from which the point cloud is being viewed and mayperform more subdivision iterations on points of the point cloud in theforeground of the point cloud as viewed from the point of view and fewersubdivision iterations on points in a background of the point cloud asviewed from the point of view.

At 618, the spatial information for the sub-sampled points of the pointcloud are encoded. Additionally, subdivision location inclusion andrelocation data is encoded. Additionally, any configurable parametersselected by the encoder or provided to the encoder from a user areencoded. The compressed point cloud may then be sent to a receivingentity as a compressed point cloud file, multiple compressed point cloudfiles, or may be packetized and communicated via multiple packets to areceiving entity, such as a decoder or a storage device. In someembodiments, a compressed point cloud may comprise both compressedspatial information and compressed attribute information. In otherembodiments, compressed spatial information and compressed attributeinformation may be included is separate compressed point cloud files.

FIG. 7 illustrates another example process for compressing spatialinformation of a point cloud, according to some embodiments.

In some embodiments, other spatial information compression techniquesother than the sub-sampling and prediction spatial information techniquedescribed in FIGS. 6A-B may be used. For example, a spatial encoder,such as spatial encoder 204, or a spatial decoder, such as spatialdecoder 222, may utilize other spatial information compressiontechniques, such as a K-D tree spatial information compressiontechnique. For example, compressing spatial information at 406 of FIG. 4may be performed using a sub-sampling and prediction technique similarto what is described in FIGS. 6A-B, may be performed using a K-D treespatial information compression technique similar to what is describedin FIG. 7 , or may be performed using another suitable spatialinformation compression technique.

In a K-D tree spatial information compression technique, a point cloudcomprising spatial information may be received at 702. In someembodiments, the spatial information may have been previously quantizedor may further be quantized after being received. For example 718illustrates a captured point cloud that may be received at 702. Forsimplicity, 718 illustrates a point cloud in two dimensions. However, insome embodiments, a received point cloud may include points in 3-Dspace.

At 704, a K-dimensional tree or K-D tree is built using the spatialinformation of the received point cloud. In some embodiments, a K-D treemay be built by dividing a space, such as a 1-D, 2-D, or 3-D space of apoint cloud in half in a predetermined order. For example, a 3-D spacecomprising points of a point cloud may initially be divided in half viaa plane intersecting one of the three axis, such as the X-axis. Asubsequent division may then divide the resulting space along anotherone of the three axis, such as the Y-axis. Another division may thendivide the resulting space along another one of the axis, such as theZ-axis. Each time a division is performed a number of points included ina child cell created by the division may be recorded. In someembodiments, only a number of points in one child cell of two childcells resulting from a division may be recorded. This is because anumber of points included in the other child cell can be determined bysubtracting the number of points in the recorded child cell from a totalnumber of points in a parent cell prior to the division.

A K-D tree may include a sequence of number of points included in cellsresulting from sequential divisions of a space comprising points of apoint cloud. In some embodiments, building a K-D tree may comprisecontinuing to subdivide a space until only a single point is included ineach lowest level child cell. A K-D tree may be communicated as asequence of number of points in sequential cells resulting fromsequential divisions. A decoder may be configured with informationindicating the subdivision sequence followed by an encoder. For example,an encoder may follow a pre-defined division sequence until only asingle point remains in each lowest level child cell. Because thedecoder may know the division sequence that was followed to build theK-D tree and the number of points that resulted from each subdivision(which is communicated to the decoder as compressed spatial information)the decoder may be able to reconstruct the point cloud.

For example, 720 illustrates a simplified example of K-D compression ina two-dimensional space. An initial space includes seven points. Thismay be considered a first parent cell and a K-D tree may be encoded witha number of points “7” as a first number of the K-D tree indicating thatthere are seven total points in the K-D tree. A next step may be todivide the space along the X-axis resulting in two child cells, a leftchild cell with three points and a right child cell with four points.The K-D tree may include the number of points in the left child cell,for example “3” as a next number of the K-D tree. Recall that the numberof points in the right child cell can be determined based on subtractingthe number of points in the left child cell from the number of points inthe parent cell. A further step may be to divide the space an additionaltime along the Y-axis such that each of the left and right child cellsare divided in half into lower level child cells. Again, a number ofpoints included in the left lower-level child cells may be included in aK-D tree, for example “0” and “1”. A next step may then be to divide thenon-zero lower-level child cells along the X-axis and record the numberof points in each of the lower-level left child cells in a K-D tree.This process may continue until only a single point remains in a lowestlevel child cell. A decoder may utilize a reverse process to recreate apoint cloud based on receiving a sequence of point totals for each leftchild cell of a K-D tree.

At 706, an encoding context for encoding a number of points for a firstcell of the K-D tree, for example the parent cell comprising sevenpoints, is selected. In some embodiments, a context store may storehundreds or thousands of encoding contexts. In some embodiments, cellscomprising more points than a highest number of points encoding contextmay be encoded using the highest number point encoding context. In someembodiments, an encoding context may include arithmetic encoding, Golombexponential encoding, or a combination of the two. In some embodiments,other encoding techniques may be used. In some embodiments, anarithmetic encoding context may include probabilities for particularsymbols, wherein different arithmetic encoding contexts includedifferent symbol probabilities.

At 708, the number of points for the first cell is encoded according theselected encoding context.

At 710, an encoding context for encoding a child cell is selected basedon a number of points included in a parent cell. The encoding contextfor the child cell may be selected in a similar manner as for the parentcell at 706.

At 712, the number of points included in the child cell is encodedaccording the selected encoding context, selected at 710. At 714, it isdetermined if there are additional lower-level child cells to encode inthe K-D tree. If so, the process reverts to 710. If not, at 716, theencoded number of points in the parent cell and the child cells areincluded in a compressed spatial information file, such as a compressedpoint cloud. The encoded values are ordered in the compressed spatialinformation file such that the decoder may reconstruct the point cloudbased on the number of points of each parent and child cell and theorder in which the number of points of the respective cells are includedin the compressed spatial information file.

In some embodiments, the number of points in each cell may be determinedand subsequently encoded as a group at 716. Or, in some embodiments, anumber of points in a cell may be encoded subsequent to being determinedwithout waiting for all child cell point totals to be determined.

Example Decoding Process

FIG. 8 illustrates an example process for decompressing compressedattribute information of a point cloud, according to some embodiments.

At 802, a decoder receives compressed attribute information for a pointcloud, and at 804, the decoder receives compressed spatial informationfor the point cloud. In some embodiments, the compressed attributeinformation and the compressed spatial information may be included inone or more common files or separate files.

At 806, the decoder decompresses the compressed spatial information. Thecompressed spatial information may have been compressed according to asub-sampling and prediction technique and the decoder may performsimilar sub-sampling, prediction, and prediction correction actions aswere performed at the encoder and further apply correction values to thepredicted point locations, to generate a non-compressed point cloud fromthe compressed spatial information. In some embodiments, the compressedspatial information may be compressed in a K-D tree format, and thedecoder may generate a decompressed point cloud based on an encoded K-Dtree included in the received spatial information. In some embodiments,the compressed spatial information may have been compressed using anOctree technique and an Octree decoding technique may be used togenerate decompressed spatial information for the point cloud. In someembodiments, other spatial information compression techniques may havebeen used and may be decompressed via the decoder.

At 808, the decoder an order of the points of the point cloud based on aspace-filling curve. For example, compressed spatial information and/orcompressed attribute information may be received via a encoded datainterface of a decoder, such as encoded data interface 226 of decoder220 illustrated in FIG. 2B. A spatial decoder, such as spatial decoder222, may decompress the compressed spatial information, and a spacefiling cure order generator, such as space filling curve order generator228, may generate a space filing curve order based on the decompressedspatial information.

At 810, a prediction evaluator of a decoder, such as predictionevaluator 224 of decoder 220, may assign an attribute value to astarting point based on an assigned attribute value included in thecompressed attribute information. In some embodiments, the compressedattribute information may identify a point as a starting point to beused for generating the space filling curve order and for predictingattribute values of the points according to an evaluation order based onthe space filling curve. The assigned attribute value or values for thestarting point may be included in decompressed attribute information fora decompressed point cloud.

At 812, the prediction evaluator of the decoder or another decodercomponent determines an evaluation order for at least the next pointsubsequent to the starting point that is to be evaluated. In someembodiments, an evaluation order may be determined for all or multipleones of the points, or in other embodiments, an evaluation order may bedetermined point by point as attribute values are determined for thepoints. The points may be evaluated in an order based on minimumdistances between successive points being evaluated. For example, aneighboring point at a shortest distance from a starting point ascompared to other neighboring points may be selected as a next point toevaluate subsequent to the starting point. In a similar manner, otherpoints may then be selected to be evaluated based on a shortest distancefrom a point that has most recently been evaluated. At 814, the nextpoint to evaluate is selected. In some embodiments 812 and 814 may beperformed together.

At 816, a prediction evaluator of a decoder determines the “K” nearestneighboring points to a point being evaluated. In some embodiments,neighboring points may only be included in the “K” nearest neighboringpoints if they already have assigned or predicted attribute values. Inother embodiments, neighboring points may be included in the “K” nearestneighboring points without regard to whether they have assigned oralready predicted attribute values. In such embodiments, an encoder mayfollow a similar rule as the decoder as to whether or not to includepoints without predicted values as neighboring points when identifyingthe “K” nearest neighboring points.

At 818, predicted attribute values are determined for one or moreattributes of the point being evaluated based on attribute values of the“K” nearest neighboring points and distances between the point beingevaluated and respective ones of the “K” nearest neighboring points. Insome embodiments, an inverse distance interpolation technique may beused to predict attribute values, wherein attribute values of pointscloser to a point being evaluated are weighted more heavily thanattribute values of points that are further away from the point beingevaluated. The attribute prediction technique used by a decoder may bethe same as an attribute prediction technique used by an encoder thatcompressed the attribute information.

At 820, a prediction evaluator of a decoder may apply an attributecorrection value to a predicted attribute value of a point to correctthe attribute value. The attribute correction value may cause theattribute value to match or nearly match an attribute value of anoriginal point cloud prior to compression. In some embodiments, in whicha point has more than one attribute, 818 and 820 may be repeated foreach attribute of the point. In some embodiments, some attributeinformation may be decompressed without decompressing all attributeinformation for a point cloud or a point. For example, a point mayinclude velocity attribute information and color attribute information.The velocity attribute information may be decoded without decoding thecolor attribute information and vice versa. In some embodiments, anapplication utilizing the compressed attribute information may indicatewhat attributes are to be decompressed for a point cloud.

At 822, it is determined if there are additional points to evaluate. Ifso, the process reverts to 814 and a next point to evaluate is selected.If there are not additional points to evaluate, at 824, decompressedattribute information is provided, for example as a decompressed pointcloud, wherein each point comprises spatial information and one or moreattributes.

In some embodiments, a decoder may execute a complementary adaptiveprediction process as described above for an encoder in FIG. 4B. Forexample, FIG. 8B illustrates predicting attribute values as part ofdecompressing attribute information of a point cloud using adaptivedistance based prediction, according to some embodiments.

At 850, a decoder identifies a set of neighboring points for aneighborhood of a point of a point cloud for which an attribute value isbeing predicted. In some embodiments, the set of neighboring points ofthe neighborhood may be identified using a K-nearest neighbor techniqueas described herein. In some embodiments, points to be used to determinevariability may be identified in other manners. For example, in someembodiments, a neighborhood of points used for variability analysis maybe defined to include more or fewer points or points within a greater orsmaller distance from the given point than are used to predict attributevalues based on inverse distance based interpolation using the K-nearestneighboring points. In some embodiments, wherein parameters used toidentify the neighborhood points for determining variability differ fromthe parameters used in a K-nearest neighbor prediction, the differingparameters or data from which the differing parameter may be determinedis signaled in a bit stream encoded by an encoder and received at thedecoder.

At 852, the variability of the attribute values of the neighboringpoints is determined. In some embodiment, each attribute valuevariability may be determined separately. For example, for points withR, G, B attribute values each attribute value (e.g. each of R, G, and B)may have their respective variabilities determined separately. Also, insome embodiments trellis quantization may be used wherein a set ofattributes such as RGB that have correlated values may be determined asa common variability. For example, in the example discussed above withregard to the white stripe on the black road, the large variability in Rmay also apply to B and G, thus it is not necessary to determinevariability for each of R, G, and B separately. Instead the relatedattribute values can be considered as a group and a common variabilityfor the correlated attributes can be determined.

In some embodiments, the variability of the attributes in theneighborhood of point P may be determined using: a sum of square errorsvariability technique, a distance weighted sum of square errorsvariability technique, a sum of absolute differences variabilitytechnique, a distance weighted sum of absolute differences variabilitytechnique, or other suitable variability technique. In some embodimentsthe decoder may utilize a variability technique to signaled be used fora given point P. In some embodiments, the decoder may determine whichvariability technique to use based on an index value encoded in the bitstream, wherein the index value is for an index of variabilitytechniques, wherein the decoder includes the same index as the encoderand can determine which variability technique to use for point P basedon the encoded index value.

At 854 through 856 it is determined whether or not the variabilitydetermined at 852 exceeds one or more variability thresholds. If so, acorresponding prediction technique that corresponds with the exceededvariability threshold is used to predict the attribute value or valuesfor the point P. In some embodiments, multiple prediction procedures maybe supported. For example, element 858 indicates using a firstprediction procedure if a first variability threshold is exceeded andelement 860 indicates using another prediction procedure if anothervariability threshold is exceeded. Furthermore, 862 indicates using adefault prediction procedure, such as a non-modified K-nearest neighborprediction procedure if the variability thresholds 1 through N are notexceeded. In some embodiments, a single variability threshold and asingle alternate prediction procedure may be used in addition to adefault prediction procedure. In some embodiments, any number of “N”variability thresholds and corresponding prediction procedures may beused.

Level of Detail Attribute Compression

In some circumstances, a number of bits needed to encode attributeinformation for a point cloud may make up a significant portion of a bitstream for the point cloud. For example, the attribute information maymake up a larger portion of the bit stream than is used to transmitcompressed spatial information for the point cloud.

In some embodiments, spatial information may be used to build ahierarchical Level of Detail (LOD) structure. The LOD structure may beused to compress attributes associated with a point cloud. The LODstructure may also enable advanced functionalities such asprogressive/view-dependent streaming and scalable rendering. Forexample, in some embodiments, compressed attribute information may besent (or decoded) for only a portion of the point cloud (e.g. a level ofdetail) without sending (or decoding) all of the attribute informationfor the whole point cloud.

FIG. 9 illustrates an example encoding process that generates ahierarchical LOD structure, according to some embodiments. For example,in some embodiments, an encoder such as encoder 202 may generatecompressed attribute information in a LOD structure using a similarprocess as shown in FIG. 9 .

In some embodiments, geometry information (also referred to herein as“spatial information”) may be used to efficiently predict attributeinformation. For example, in FIG. 9 the compression of color informationis illustrated. However, a LOD structure may be applied to compressionof any type of attribute (e.g., reflectance, texture, modality, etc.)associated with points of a point cloud. Note that a pre-encoding stepwhich applies color space conversion or updates the data to make thedata better suited for compression may be performed depending on theattribute to be compressed.

In some embodiments, attribute information compression according to aLOD process proceeds as described below.

For example, let Geometry (G)={Point-P(0), P(1), . . . P(N−1)} bereconstructed point cloud positions generated by a spatial decoderincluded in an encoder (geometry decoder GD 902) after decoding acompressed geometry bit stream produced by a geometry encoder, alsoincluded in the encoder (geometry encoder GE 914), such as spatialencoder 204 (illustrated in FIG. 2A). For example, in some embodiments,an encoder such as encoder 202 (illustrated in FIG. 2A) may include botha geometry encoder, such as geometry encoder 914, and a geometrydecoder, such as geometry decoder 902. In some embodiments, a geometryencoder may be part of spatial encoder 214 and a geometry decoder may bepart of prediction/correction evaluator 206, both as illustrated in FIG.2A.

In some embodiments, the decompressed spatial information may describelocations of points in 3D space, such as X, Y, and Z coordinates of thepoints that make up mug 900. Note that spatial information may beavailable to both an encoder, such as encoder 202, and a decoder, suchas decoder 220. For example various techniques, such as K-D treecompression, octree compression, nearest neighbor prediction, etc., maybe used to compress and/or encode spatial information for mug 900 andthe spatial information may be sent to a decoder with, or in additionto, compressed attribute information for attributes of the points thatmake up a point cloud, such as a point cloud for mug 900.

In some embodiments, a deterministic re-ordering process may be appliedon both an encoder side (such as at encoder 202) and at a decoder side(such as at decoder 220) in order to organize points of a point cloud,such as the points that represent mug 900, into a set of Level ofDetails (LODs). For example, levels of detail may be generated by alevel of detail generator 904, which may be included in aprediction/correction evaluator of an encoder, such asprediction/correction evaluator 206 of encoder 202 as illustrated inFIG. 2A. In some embodiments, a level of detail generator 904 may be aseparate component of an encoder, such as encoder 202. For example,level of detail generator 904 may be a separate component of encoder202. Note that, in some embodiments, no additional information needs tobe included in the bit stream to generate such LOD structures, exceptfor the parameters of the LOD generation algorithm. For example,parameters that may be included in a bit stream as parameters of the LODgenerator algorithm may include:

-   -   i. The maximum number of LODs to be generated denoted by “N”        (e.g., N=6),    -   ii. The initial sampling distance “D0” (e.g., D0=64), and    -   iii. The sampling distance update factor “f” (e.g., ½).

In some embodiments, the parameters N, D0 and f, may be provided by auser, such as an engineer configuring a compression process. In someembodiments the parameters N, D0 and f, may be determined automaticallyby an encoder/and or decoder using an optimization procedure, forexample. These parameters may be fixed or adaptive.

In some embodiments, LOD generation may proceed as follows:

-   -   a. Points of geometry G (e.g. the points of the point cloud        organized according to the spatial information), such as points        of mug 900, are marked as non-visited and a set of visited        points V is set to be empty.    -   b. The LOD generation process may then proceed iteratively. At        each iteration j, the level of detail for that refinement level,        e.g. LOD(j), may be generated as follows:        -   1. The sampling distance for the current LOD, denoted D(j)            may be set as follows:            -   a. If j=0, then D(j)=D0.            -   b. If j>0 and j<N, then D(j)=D(j−1)*f.            -   c. if j=N, then D(j)=0.        -   2. The LOD generation process iterates over all the points            of G.            -   a. At the point evaluation iteration i, a point P(i) is                evaluated,                -   i. if the point P(i) has been visited then it is                    ignored and the algorithm jumps to the next                    iteration (i+1), e.g. the next point P(i+1) is                    evaluated.                -   ii. Otherwise, the distance D(i, V), defined as the                    minimum distance from P(i) over all the points of V,                    is computed. Note that V is the list of points that                    have already been visited. If V is empty, the                    distance D(i, V) is set to 0, meaning that the                    distance from point P(i) to the visited points is                    zero because there are not any visited points in the                    set V. If the shortest distance from point P(i) to                    any of the already visited point, D(i, V), is                    strictly higher than a parameter D0, then the point                    is ignored and the LoD generation jumps to the                    iteration (i+1) and evaluates the next point P(i+1).                    Otherwise, P(i) is marked as a visited point and the                    point P(i) is added to the set of visited points V.            -   b. This process may be repeated until all the points of                geometry G are traversed.        -   3. The set of points added to V during the iteration j            describes the refinement level R(j).        -   4. The LOD(j) may be obtained by taking the union of all the            refinement levels R(0), R(1), . . . , R(j).

In some embodiments, the process described above, may be repeated untilall the LODs are generated or all the vertices have been visited.

In some embodiments, an encoder as described above may further include aquantization module (not shown) that quantizes geometry informationincluded in the “positions (x,y,z) being provided to the geometryencoder 914. Furthermore, in some embodiments, an encoder as describedabove may additionally include a module that removes duplicated pointssubsequent to quantization and before the geometry encoder 914.

In some embodiments, quantization may further be applied to compressedattribute information, such as attribute correction values and/or one ormore attribute value starting points. For example quantization isperformed at 910 to attribute correction values determined byinterpolation-based prediction module 908. Quantization techniques mayinclude uniform quantization, uniform quantization with a dead zone,non-uniform/non-linear quantization, trellis quantization, or othersuitable quantization techniques.

FIG. 10 illustrates an example process for determining points to beincluded at different refinement layers of a level of detail (LOD)structure, according to some embodiments.

At 1002 an encoder (or a decoder) receives or determines level of detailparameters to use in determining the level of detail hierarchy for thepoint cloud. At the same time, or before or after, receiving the levelof detail parameters, at 1004 the encoder (or decoder) may receivecompressed spatial information for the point cloud and at 1006, theencoder (or decoder) may determine decompressed spatial information forthe points of the point cloud. In embodiments that utilize lossycompression techniques for the compression of the spatial information,the compressed spatial information may be compressed at the encoder andmay also be decompressed at the encoder at 1006 to generate arepresentative sample of the geometry information that will beencountered at a decoder. In some embodiments that utilize losslesscompression of spatial information, 1004 and 1106 may be omitted on theencoder side.

At 1008 the level of detail structure generator (which may be on anencoder side or a decoder side) marks all the points of the point cloudas “non-visited points.”

At 1010 the level of detail structure generator also sets a directory ofvisited point “V” to be empty.

At 1012 a sampling distance D(j) is determined for a current level ofrefinement being evaluated, R(j). If the level of refinement is thecoarsest level of refinement, where j=0, the sampling distance D(j) isset to be equal to D0, e.g. the initial sampling distance, which wasreceived or determined at 1002. If j is greater than 0, but less than N,then D(j) is set to equal to D(j−1)*f. Note that “N” is the total numberof level of details that are to be determined. Also note that “f” is asampling update distance factor which is set to be less than one (e.g.½). Also, note that D(j−1) is the sampling distance used in thepreceding level of refinement. For example, when f is ½, the samplingdistance D(j−1) is cut in half for a subsequent level of refinement,such that D(j) is one half the length of D(j−1). Also, note that a levelof detail (LOD(j)) is the union of a current level of refinement and alllower levels of refinement. Thus, a first level of detail (LOD(0)) mayinclude all the points included in level of refinement R(0). Asubsequent level of detail (LOD(1) may include all of the pointsincluded in the previous level of detail and additionally all the pointsincluded in the subsequent level of refinement R(1). In this way pointsmay be added to a previous level of detail for each subsequent level ofdetail until a level of detail “N” is reached that includes all of thepoints of the point cloud.

To determine the points of the point cloud to be included in a currentlevel of detail which is being determined, a point P(i) is selected tobe evaluated at 1014, where “i” is a current one of the points of thepoint cloud that is being evaluated. For example if a point cloudincludes a million points, “i” could range from 0 to 1,000,000.

At 1016, it is determined if the point currently being evaluated, P(i),has already been marked as a visited point. If P(i) is marked as alreadyvisited, then at 1018 P(i) is ignored and the process moves on toevaluate the next point P(i+1), which then becomes the point currentlybeing evaluated P(i). The process then reverts back to 1014.

If it is determined at 1016, that point P(i) has not already been markedas a visited point, at 1020 a distance D(i) is computed for the pointP(i), where D(i) is the shortest distance between point P(i) and any ofthe already visited points included in directory V. If there are not anypoints included in directory V, e.g. the directory V is empty, then D(i)is set to zero.

At 1022, it is determined whether the distance D(i) for point P(i) isgreater than the initial sampling distance D0. If so, at 1018 point P(i)is ignored and the process moves on to the next point P(i+1) and revertsto 1014.

If point P(i) is not already marked as visited and the distance D(i),which is a minimum distance between point P(i) and the set of pointsincluded in V, is less than the initial sampling distance D0, then at1024 point P(i) is marked as visited and added to a set of visitedpoints V for the current level of refinement R(j).

At 1026, it is determined if there are additional levels of refinementto determine for the point cloud. For example, if j<N, then there areadditional levels of refinement to determine, where N is a LOD parameterthat may be communicated between an encoder and decoder. If there arenot additional levels of refinement to determine, the process stops at1028. If there are additional levels of refinement to determine theprocess moves on to the next level of refinement at 1030, and thenproceeds to evaluate the point cloud for the next level of refinement at1012.

Once the levels of refinement have been determined, the levels ofrefinement may be used generate the LOD structure, where each subsequentLOD level includes all the points of a previous LOD level plus anypoints determined to be included in an additional level of refinement.Because the process for determining an LOD structure is known by theencoder and decoder, a decoder, given the same LOD parameters as used atan encoder, can recreate the same LOD structure at the decoder as wasgenerated at the encoder.

Example Level of Detail Hierarchy

FIG. 11A illustrates an example LOD, according to some embodiments. Notethat the LOD generation process may generate uniformly sampledapproximations (or levels of detail) of the original point cloud, thatget refined as more and more points are included. Such a feature makesit particularly adapted for progressive/view-dependent transmission andscalable rendering. For example, 1104 may include more detail than 1102,and 1106 may include more detail than 1104. Also, 1108 may include moredetail than 1102, 1104, and 1106.

The hierarchical LOD structure may be used to build an attributeprediction strategy. For example, in some embodiments the points may beencoded in the same order as they were visited during the LOD generationphase. Attributes of each point may be predicted by using the K-nearestneighbors that have been previously encoded. In some embodiments, “K” isa parameter that may be defined by the user or may be determined byusing an optimization strategy. “K” may be static or adaptive. In thelatter case where “K” is adaptive, extra information describing theparameter may be included in the bit stream.

In some embodiments, different prediction strategies may be used. Forexample, one of the following interpolation strategies may be used, aswell as combinations of the following interpolation strategies, or anencoder/decoder may adaptively switch between the differentinterpolation strategies. The different interpolation strategies mayinclude interpolation strategies such as: inverse-distanceinterpolation, barycentric interpolation, natural neighborinterpolation, moving least squares interpolation, or other suitableinterpolation techniques. For example, interpolation based predictionmay be performed at an interpolation-based prediction module 908included in a prediction/correction value evaluator of an encoder, suchas prediction/correction value evaluator 206 of encoder 202. Also,interpolation based prediction may be performed at aninterpolation-based prediction module 908 included in a predictionevaluator of a decoder, such as prediction evaluator 224 of decoder 220.In some embodiments, a color space may also be converted, at color spaceconversion module 906, prior to performing interpolation basedprediction. In some embodiments, a color space conversion module 906 maybe included in an encoder, such as encoder 202. In some embodiments, adecoder may further included a module to convert a converted colorspace, back to an original color space.

In some embodiments, quantization may further be applied to attributeinformation. For example quantization may performed at quantizationmodule 910. In some embodiments, a encoder, such as encoder 202, mayfurther include a quantization module 910. Quantization techniquesemployed by a quantization module 910 may include uniform quantization,uniform quantization with a dead zone, non-uniform/non-linearquantization, trellis quantization, or other suitable quantizationtechniques.

In some embodiments, LOD attribute compression may be used to compressdynamic point clouds as follows:

-   -   a. Let FC be the current point cloud frame and RF be the        reference point cloud.    -   b. Let M be the motion field that deforms RF to take the shape        of FC.        -   i. M may be computed on the decoder side and in this case            information may not be encoded in the bit stream.        -   ii. M may be computed by the encoder and explicitly encoded            in the bit stream            -   1. M may be encoded by applying a hierarchical                compression technique as described herein to the motion                vectors associated with each point of RF (e.g. the                motion of RF may be considered as an extra attribute).            -   2. M may be encoded as a skeleton/skinning-based model                with associated local and global transforms.            -   3. M may be encoded as a motion field defined based on                an octree structure, which is adaptively refined to                adapt to motion field complexity.            -   4. M may be described by using any suitable animation                technique such as key-frame-based animations, morphing                techniques, free-form deformations, key-point-based                deformation, etc.        -   iii. Let RF′ be the point cloud obtained after applying the            motion field M to RF. The points of RF′ may be then used in            the attribute prediction strategy by considering not only            the “K” nearest neighbor points of FC but also those of RF′.

Furthermore, attribute correction values may be determined based oncomparing the interpolation-based prediction values determined atinterpolation-based prediction module 908 to original non-compressedattribute values. The attribute correction values may further bequantized at quantization module 910 and the quantitated attributecorrection values, encoded spatial information (output from the geometryencoder 902) and any configuration parameters used in the prediction maybe encoded at arithmetic encoding module 912. In some embodiments, thearithmetic encoding module, may use a context adaptive arithmeticencoding technique. The compressed point cloud may then be provided to adecoder, such as decoder 220, and the decoder may determine similarlevels of detail and perform interpolation based prediction to recreatethe original point cloud based on the quantized attribute correctionvalues, encoded spatial information (output from the geometry encoder902) and the configuration parameters used in the prediction at theencoder.

FIG. 11B illustrates an example compressed point cloud file comprisingLODs, according to some embodiments. Level of detail attributeinformation file 1150 includes configuration information 1152, pointcloud data 1154, and level of detail point attribute correction values1156. In some embodiments, level of detail attribute information file1150 may be communicated in parts via multiple packets. In someembodiments, not all of the sections shown in the level of detailattribute information file 1150 may be included in each packettransmitting compressed attribute information. In some embodiments, alevel of detail attribute information file, such as level of detailattribute information file 1150, may be stored in a storage device, suchas a server that implements an encoder or decoder, or other computingdevice.

FIG. 12A illustrates a method of encoding attribute information of apoint cloud using an update operation, according to some embodiments.

At 1202, a point cloud is received by an encoder. The point cloud may becaptured, for example by one or more sensors, or may be generated, forexample in software.

At 1204, spatial or geometry information of the point cloud is encodedas described herein. For example, the spatial information may be encodedusing K-D trees, Octrees, a neighbor prediction strategy, or othersuitable technique to encode the spatial information.

At 1206, one or more level of details are generated, as describedherein. For example, the levels of detail may be generated using asimilar process as shown in FIG. 10 . Note that in some embodiments, thespatial information encoded or compressed at 1204 may be de-coded ordecompressed to generate a representative decompressed point cloudgeometry that a decoder would encounter. This representativedecompressed point cloud geometry may then be used to generate a LODstructure as further described in FIG. 10 .

At 1208, an interpolation based prediction is performed to predictattribute values for the attributes of the points of the point cloud. At1210, attribute correction values are determined based on comparing thepredicted attribute values to original attribute values. For example, insome embodiments, an interpolation based prediction may be performed foreach level of detail to determine predicted attribute values for pointsincluded in the respective levels of detail. These predicted attributevalues may then be compared to attribute values of the original pointcloud prior to compression to determine attribute correction values forthe points of the respective levels of detail. For example, aninterpolation based prediction process as described in FIG. 1B, FIGS.4-5 , and FIG. 8 may be used to determine predicted attribute values forvarious levels of detail. In some embodiments, attribute correctionvalues may be determined for multiple levels of detail of a LODstructure. For example a first set of attribute correction values may bedetermined for points included in a first level of detail and additionalsets of attribute correction values may be determined for pointsincluded in other levels of detail.

At 1212, an update operation may optionally be applied that affects theattribute correction values predicted at 1210. Performance of the updateoperation is discussed in more detail below in FIG. 13A.

At 1214, attribute correction values, LOD parameters, encoded spatialinformation (output from the geometry encoder) and any configurationparameters used in the prediction are encoded, as described herein.

In some embodiments, the attribute information encoded at 1214 mayinclude attribute information for multiple or all levels of detail ofthe point cloud, or may include attribute information for a single levelof detail or fewer than all levels of detail of the point cloud. In someembodiments, level of detail attribute information may be sequentiallyencoded by an encoder. For example, an encoder may make available afirst level of detail before encoding attribute information for one ormore additional levels of detail.

In some embodiments, an encoder may further encode one or moreconfiguration parameters to be sent to a decoder, such as any of theconfiguration parameters shown in configuration information 1152 ofcompressed attribute information file 1150. For example, in someembodiments, an encoder may encode a number of levels of detail that areto be encoded for a point cloud. The encoder may also encode a samplingdistance update factor, wherein the sampling distance is used todetermine which points are to be included in a given level of detail.

FIG. 12B illustrates a method of decoding attribute information of apoint cloud, according to some embodiments.

At 1252, compressed attribute information for a point cloud is receivedat a decoder. Also, at 1254 spatial information for the point cloud isreceived at the decoder. In some embodiments, the spatial informationmay be compressed or encoded using various techniques, such as a K-Dtree, Octree, neighbor prediction, etc. and the decoder may decompressand/or decode the received spatial information at 1254.

At 1256, the decoder determines which level of detail of a number oflevels of detail to decompress/decode. The selected level of detail todecompress/decode may be determined based on a viewing mode of the pointcloud. For example, a point cloud being viewed in a preview mode mayrequire a lower level of detail to be determined than a point cloudbeing viewed in a full view mode. Also, a location of a point cloud in aview being rendered may be used to determine a level of detail todecompress/decode. For example, a point cloud may represent an objectsuch as the coffee mug shown in FIG. 9 . If the coffee mug is in aforeground of a view being rendered a higher level of detail may bedetermined for the coffee mug. However, if the coffee mug is in thebackground of a view being rendered, a lower level of detail may bedetermined for the coffee mug. In some embodiments, a level of detail todetermine for a point cloud may be determined based on a data budgetallocated for the point cloud.

At 1258 points included in the first level of detail (or next level ofdetail) being determined may be determined as described herein. For thepoints of the level of detail being evaluated, attribute values of thepoints may be predicted based on an inverse distance weightedinterpolation based on the k-nearest neighbors to each point beingevaluated, where k may be a fixed or adjustable parameter.

At 1260, in some embodiments, an update operation may be performed onthe predicted attribute values as described in more detail in FIG. 12F.

At 1262, attribute correction values included in the compressedattribute information for the point cloud may be decoded for the currentlevel of detail being evaluated and may be applied to correct theattribute values predicted at 1258 or the updated predicted attributevalues determined at 1260.

At 1264, the corrected attribute values determined at 1262 may beassigned as attributes to the points of the first level of detail (orthe current level of detail being evaluated). In some embodiments, theattribute values determined for subsequent levels of details may beassigned to points included in the subsequent levels of detail whileattribute values already determined for previous levels of detail areretained by the respective points of the previous level(s) of detail. Insome embodiments, new attribute values may be determined for sequentiallevels of detail.

In some embodiments, the spatial information received at 1254 mayinclude spatial information for multiple or all levels of detail of thepoint cloud, or may include spatial information for a single level ofdetail or fewer than all levels of detail of the point cloud. In someembodiments, level of detail attribute information may be sequentiallyreceived by a decoder. For example, a decoder may receive a first levelof detail and generate attribute values for points of the first level ofdetail before receiving attribute information for one or more additionallevels of detail.

At 1266 it is determined if there are additional levels of detail todecode. If so, the process returns to 1258 and is repeated for the nextlevel of detail to decode. If not the process is stopped at 1267, butmay resume at 1256 in response to input affecting the number of levelsof detail to determine, such as change in view of a point cloud or azoom operation being applied to a point cloud being viewed, as a fewexamples of an input affecting the levels of detail to be determined.

In some embodiments the spatial information described above may beencoded and decoded via a geometry encoder and arithmetic encoder, suchas geometry encoder 202 and arithmetic encoder 212 described above inregard to FIG. 2 . In some embodiments, a geometry encoder, such asgeometry encoder 202 may utilize an octree compression technique andarithmetic encoder 212 may be a binary arithmetic encoder as describedin more detail below.

The use of a binary arithmetic encoder as described below reduces thecomputational complexity of encoding octree occupancy symbols ascompared to a multi-symbol codec with an alphabet of 256 symbols (e.g. 8sub-cubes per cube, and each sub-cube occupied or un-occupied2{circumflex over ( )}8=256). Also the use of context selection based onmost probable neighbor configurations may reduce a search for neighborconfigurations, as compared to searching all possible neighborconfigurations. For example, the encoder may keep track of 10 encodingcontexts which correspond to the 10 neighborhood configurations 1268,1270, 1272, 12712, 1276, 1278, 1280, 1282, 1284, and 1286 shown in FIG.12C as opposed to all possible neighborhood configurations.

In some embodiments, an arithmetic encoder, such as arithmetic encoder212, may use a binary arithmetic codec to encode the 256-value occupancysymbols. This may be less complex and more hardware friendly in terms ofimplementation as compared to a multi-symbol arithmetic codec.Additionally, an arithmetic encoder 212 and/or geometry encoder 202 mayutilizes a look-ahead procedure to compute the 6-neighbors used forarithmetic context selection, which may be less complex than a linearsearch and may involve a constant number of operations (as compared to alinear search which may involve varying numbers of operations).Additionally, the arithmetic encoder 212 and/or geometry encoder 202 mayutilize a context selection procedure, which reduces the number ofencoding contexts. In some embodiments, a binary arithmetic codec,look-ahead procedure, and context selection procedure may be implementedtogether or independently.

Binary Arithmetic Encoding

In some embodiments, to encode spatial information, occupancyinformation per cube is encoded as an 8-bit value that may have a valuebetween 0-255. To perform efficient encoding/decoding of such non-binaryvalues, typically a multi-symbol arithmetic encoder/decoder would beused, which is computationally complex and less hardware friendly toimplement when compared to a binary arithmetic encoder/decoder. However,direct use of a conventional binary arithmetic encoder/decoder on such avalue on the other hand, e.g. encoding each bit independently, may notbe as efficient. However, in order, to efficiently encode the non-binaryoccupancy values with a binary arithmetic encoder an adaptive look uptable (A-LUT), which keeps track of the N (e.g., 32) most frequentoccupancy symbols, may be used along with a cache which keeps track ofthe last different observed M (e.g., 16) occupancy symbols.

The values for the number of last different observed occupancy symbols Mto track and the number of the most frequent occupancy symbols N totrack may be defined by a user, such as an engineer customizing theencoding technique for a particular application, or may be chosen basedon an offline statistical analysis of encoding sessions. The choice ofthe values of M and N may be based on a compromise between:

-   -   Encoding efficiency,    -   Computational complexity, and    -   Memory requirements.

In some embodiments, the algorithm proceeds as follows:

-   -   The adaptive look-up table (A-LUT) is initialized with N symbols        provided by the user (e.g. engineer) or computed offline based        on the statistics of a similar class of point clouds.    -   The cache is initialized with M symbols provided by the user        (e.g. engineer) or computed offline based on the statistics of a        similar class of point clouds.    -   Every time an occupancy symbol S is encoded the following steps        are applied        -   1. A binary information indicating whether S is in the A-LUT            or not is encoded.        -   2. If S is in the A-LUT, the index of S in the A-LUT is            encoded by using a binary arithmetic encoder            -   Let (b1, b2, b3, b4, b5) be the five bits of the binary                representation of the index of S in the A-LUT. Let b1 be                the least significant bit and b5 the most significant                bit.            -   Three approaches as described below to encode the index                of S may be used, for example by using either 31, 9, or                5 adaptive binary arithmetic contexts as shown below                -   31 Contexts                -    First encode b5 of the index of S with a first                    context (call it context 0), when encoding the most                    significant bit (the first bit to be encoded) there                    is not any information that can be used from the                    encoding of other bits, that is why the context is                    referred to as context zero. Then when encoding b4                    (the second bit to be encoded), there are two                    additional contexts that may be used call them                    context 1 (if b5=0) and context 2 (if b5=1). When                    this approach is taken all the way out to b1, there                    are 31 resulting contexts as shown in the diagram                    below, context 0-30. This approach exhaustively uses                    each bit that is encoded to select an adaptive                    context for encoding the next bit. For example, see                    FIG. 12E.                -   9 Contexts                -    Keep in mind that the index values of the adaptive                    look-up table ALUT are assigned based on how                    frequently the symbol S has appeared. Thus the most                    frequent symbol S in the ALUT would have an index                    value of 0 meaning that all of the bits of the index                    value for the most frequent symbol S are zero. For                    example, the smaller the binary value, the more                    frequently the symbol has appeared. To encode nine                    contexts, for b4 and b5, which are the most                    significant bits, if they are is the index value                    must be comparatively large. For example if b5=1                    then the index value is at least 16 or higher, or if                    b4=1 the index value is at least 8 or higher. So                    when encoding 9 contexts, the focus is placed on the                    first 7 index entries, for example 1-7. For these 7                    index entries adaptive encoding contexts are used.                    However for index entries with values greater than 7                    the same context is used, for example a static                    binary encoder. Thus, if b5=1 or b4=1, then the same                    context is used to encode the index value. If not,                    then one of the adaptive contexts 1-7 is used.                    Because there is a context 0 for b5, 7 adaptive                    contexts, and a common context for entries strictly                    greater than 8, there are nine total contexts. This                    simplifies encoding and reduces the number of                    contexts to be communicated as compared to using all                    31 contexts as shown above.                -   5 contexts                -    To encode an index value using 5 contexts,                    determine if b5=1. If b5=1 then use a static binary                    context to encode all the bits of the index value                    from b4 to b1. If b5 does not equal 1, then encode                    b4 of the index value and see if b4 is equal to 1                    or 0. If b4=1, which means the index value is higher                    than 8, then again use the static binary context to                    encode the bits b3 to b1. This reasoning then                    repeats, so that if b3=1, the static binary context                    is used to encode bits b2 to b1, and if b2=1 the                    static binary context is used to encode bit 1.                    However, if bits b5, b4, and b3 are equal to zero,                    then an adaptive binary context is selected to                    encode bit 2 and bit 1 of the index value.        -   3. If S is not in the A-LUT, then            -   A binary information indicating whether S is in the                cache or not is encoded.            -   If S is in the cache, then the binary representation of                its index is encoded by using a binary arithmetic                encoder                -   In some embodiments, the binary representation of                    the index is encoded by using a single static binary                    context to encode each bit, bit by bit. The bit                    values are then shifted over by one, where the least                    significant bit becomes the next more significant                    bit.            -   Otherwise, if S is not in the cache, then the binary                representation of S is encoded by using a binary                arithmetic encoder                -   In some embodiments, the binary representation of S                    is encoded by using a single adaptive binary                    context. It is known that the index have a value                    between 0 and 255, which means it is encoded on 8                    bits. The bits are shifted so that the least                    significant bit becomes the next more significant                    bit, and a same adaptive context is used to encode                    all of the remaining bits.            -   The symbol S is added to the cache and the oldest symbol                in the cache is evicted.        -   4. The number of occurrences of the symbol S in A-LUT is            incremented by one.        -   5. The list of the N most frequent symbols in the A-LUT is            re-computed periodically            -   Approach 1: If the number of symbols encoded so far                reaches a user-defined threshold (e.g., 64 or 128), then                the list of the N most frequent symbols in the A-LUT is                re-computed.            -   Approach 2: Adapts the update cycle to the number of                symbols encoded. The idea is to update the probabilities                fast in the beginning and exponentially increase the                update cycle with the number of symbols:                -   The update cycle_updateCycle is initialized to a low                    number N0 (e.g. 16).                -   Every time the number of symbols reaches the update                    cycle                -    the list of the N most frequent symbols in the                    A-LUT is re-computed                -    The update cycle is updated as follows:                    _updateCycle=min(_alpha*_updateCycle,maxUpdateCycle)                -    _alpha (e.g., 5/4) and maxUpdateCycle (e.g., 1024)                    are two user-defined parameters, which may control                    the speed of the exponential growth and the maximum                    update cycle value.        -   6. At the start of each level of the octree subdivision, the            occurrences of all symbols are reset to zero. The            occurrences of the N most frequent symbols are set to 1.        -   7. When the occurrence of a symbol reaches a user-defined            maximum number (e.g., _maxOccurence=1024), the occurrences            of all the symbols are divided by 2 to keep the occurrences            within a user-defined range.

In some embodiments, a ring—buffer is used to keep track of the elementsin the cache. The element to be evicted from the cache corresponds tothe position index0=(_last++) % CacheSize, where _last is a counterinitialized to 0 and incremented every time a symbol is added to thecache. In some embodiments, the cache could also be implemented with anordered list, which would guarantee that every time the oldest symbol isevicted.

2. Look-Ahead to Determine Neighbors

In some embodiments, at each level of subdivision of the octree, cubesof the same size are subdivided and an occupancy code for each one isencoded.

-   -   For subdivision level 0, there may be a single cube of        (2^(C),2^(C),2^(C)) without any neighbors.    -   For subdivision level 1, there may be up to 8 cubes of dimension        (2^(C-1),2^(C-1),2^(C-1)) each.    -   . . .    -   For subdivision level L, there may be up to 8^(L) cubes of        dimension (2^(C-L),2^(C-L),2^(C-L)) each.

In some embodiments, at each level L, a set of non-overlappinglook-ahead cubes of dimension (2H-C+L,2H-C+L,2H-C+L) each may bedefined, as shown in FIG. 12D. Note that the look-ahead cube can fit23×H cubes of size (2C-L,2C-L,2C-L).

-   -   At each level L, the cubes contained in each look-ahead cube are        encoded without referencing cubes in other look-ahead cubes.        -   During the look-ahead phase, the cubes of dimension            (2^(C-L),2^(C-L),2^(C-L)) in the current look-ahead cube are            extracted from the FIFO and a look-up table that describes            for each (2^(C-L),2^(C-L),2^(C-L)) region of the current            look-ahead cube whether it is occupied or empty is filled.        -   Once, the look-up table is filled, the encode phase for the            extracted cubes begins. Here, the occupancy information for            the 6 neighbors is obtained by fetching the information            directly from the look up table.        -   For cubes on the boundary of the look-ahead cube, the            neighbors located outside are assumed to be empty.            -   Another alternative could consist in filling the values                of the outside neighbors based on extrapolation methods.        -   Efficient implementation could be achieved by            -   Storing the occupancy information of each group of 8                neighboring (2^(C-L),2^(C-L),2^(C-L)) regions on one                byte            -   Storing the occupancy bytes in a Z-order to maximize                memory cache hits                3. Context Selection

In some embodiments, to reduce the number of encoding contexts (NC) to alower number of contexts (e.g., reduced from 10 to 6), a separatecontext is assigned to each of the (NC−1) most probable neighborhoodconfigurations, and the contexts corresponding to the least probableneighborhood configurations are made to share the same context(s). Thisis done as follows:

-   -   Before starting the encoding process, initialize the occurrences        of the 10 neighborhood configurations (e.g. the 10        configurations shown in FIG. 12C):        -   Set all 10 occurrences to 0        -   Set the occurrences based on offline/online statistics or            based on user-provided information.    -   At the beginning of each subdivision level of the octree:        -   Determine the (NC−1) most probable neighborhood            configurations based on the statistics collected during the            encoding of the previous subdivision level.        -   Compute a look-up table NLUT, which maps the indexes of the            (NC−1) most probable neighborhood configurations to the            numbers 0, 1, . . . , (NC−2) and maps the indexes of the            remaining configurations to NC−1.        -   Initialize the occurrences of the 10 neighborhood            configurations to 0.            -   During the encoding:                -   Increment the occurrence of a neighborhood                    configuration by one each time such a configuration                    is encountered.                -   Use the look-up table NLUT[ ] to determine the                    context to use to encode the current occupancy                    values based on the neighborhood configuration                    index.

FIG. 12F illustrates an example octree compression technique using abinary arithmetic encoder, cache, and look-ahead table, according tosome embodiments. For example, FIG. 12F illustrates an example of theprocesses as described above. At 1288 occupancy symbols for a level ofan octree of a point cloud are determined. At 1290 an adaptivelook-ahead table with “N” symbols is initialized. At 1292, a cache with“M” symbols is initialized. At 1294, symbols for the current octreelevel are encoded using the techniques described above. At 1296, it isdetermined if additional octree levels are to be encoded. If so, theprocess continues at 1288 for the next octree level. If not, the processends at 1298 and the encoded spatial information for the point cloud ismade available for use, such as being sent to a recipient or beingstored.

Lifting Schemes for Level of Detail Compression and Decompression

In some embodiments, lifting schemes may be applied to point clouds. Forexample, as described below, a lifting scheme may be applied toirregular points. This is in contrast to other types of lifting schemesthat may be applied to images having regular points in a plane. In alifting scheme, for points in a current level of detail nearest pointsin a lower level of detail are found. These nearest points in the lowerlevel of detail are used to predict attribute values for points in ahigher level of detail. Conceptually, a graph could be made showing howpoints in lower levels of detail are used to determine attribute valuesof points in higher levels of detail. In such a conceptual view, edgescould be assigned to the graph between levels of detail, wherein thereis an edge between a point in a higher level of detail and each point inthe lower level of detail that forms a basis for the prediction of theattribute of the point at the higher level of detail. As described inmore detail below, a weight could be assigned to each of these edgesindicating a relative influence. The weight may represent an influencean attribute value of the point in the lower level of detail has on theattribute value of the points in the higher level of detail. Also,multiple edges may make a path through the levels of detail and weightsmay be assigned to the paths. In some embodiments, the influence of apath may be defined by the sum of the weights of the edges of the path.For example, equation 1 discussed further below represents such aweighting of a path.

In a lifting scheme, attribute values for low influence points may behighly quantized and attribute values for high influence points may bequantized less. In some embodiments, a balance may be reached betweenquality of a reconstructed point cloud and efficiency, wherein morequantization increases compression efficiency and less quantizationincreases quality. In some embodiments, all paths may not be evaluated.For example, some paths with little influence may not be evaluated.Also, an update operator may smooth residual differences, e.g. attributecorrection values, in order to increase compression efficiency whiletaking into account relative influence or importance of points whensmoothing the residual differences.

FIG. 13A illustrates a direct transformation that may be applied at anencoder to encode attribute information of a point could, according tosome embodiments.

In some embodiments, an encoder may utilize a direct transformation asillustrated in FIG. 13A in order to determine attribute correctionvalues that are encoded as part of a compressed point cloud. Forexample, in some embodiments a direct transformation, such asinterpolation based prediction, may be utilized to determine attributevalues as described in FIG. 12A at 1208 and to apply an update operationas described in FIG. 12A at 1212.

In some embodiments, a direct transform may receive attribute signalsfor attributes associated with points of a point cloud that is to becompressed. For example, the attributes may include color values, suchas RGB colors, or other attribute values of points in a point cloud thatis to be compressed. The geometry of the points of the point cloud to becompressed may also be known by the direct transform that receives theattribute signals. At 1302, the direct transform may include a splitoperator that splits the attribute signals 1310 for a first (or next)level of detail. For example, for a particular level of detail, such asLOD(N), comprising X number of points, a sub-sample of the attributes ofthe points, e.g. a sample comprising Y points, may comprise attributevalues for a smaller number of points than X. Said another way, thesplit operator may take as an input attributes associated with aparticular level of detail and generate a low resolution sample 1304 anda high resolution sample 1306. It should be noted that a LOD structuremay be partitioned into refinement levels, wherein subsequent levels ofrefinement include attributes for more points than underlying levels ofrefinement. A particular level of detail as described below is obtainedby taking the union of all lower level of detail refinements. Forexample, the level of detail j is obtained by taking the union of allrefinement levels R(0), R(1), . . . , R(j). It should also be noted, asdescribed above, that a compressed point cloud may have a total numberof levels of detail N, wherein R(0) is the least refinement level ofdetail and R(N) is the highest refinement level of detail for thecompressed point cloud.

At 1308, a prediction for the attribute values of the points notincluded in the low resolution sample 1304 is predicted based on thepoints included in the low resolution sample. For example, based on aninverse distance interpolation prediction technique or any of the otherprediction techniques described above. At 1312, a difference between thepredicted attribute values for the points left out of low resolutionsample 1304 is compared to the actual attribute values of the pointsleft out of the low resolution sample 1304. The comparison determinesdifferences, for respective points, between a predicted attribute valueand an actual attribute value. These differences (D(N)) are then encodedas attribute correction values for the attributes of the points includedin the particular level of detail that are not encoded in the lowresolution sample. For example, for the highest level of detail N, thedifferences D(N) may be used to adjust/correct attribute values includedin lower levels of detail). Because at the highest level of detail, theattribute correction values are not used to determine attribute valuesof other even higher levels of detail (because for the highest level ofdetail, N, there are not any higher levels of detail), an updateoperation to account for relative importance of these attributecorrection values may not be performed. As such, the differences D(N)may be used to encode attribute correction values for LOD(N).

In addition, the direct transform may be applied for subsequent lowerlevels of detail, such as LOD(N−1). However, before applying the directtransform for the subsequent level of detail, an update operation may beperformed in order to determine the relative importance of attributevalues for points of the lower level of detail on attribute values ofone or more upper levels of detail. For example, update operation 1314may determine relative importances of attribute values of attributes forpoints included in lower levels of detail on higher levels of detail,such as for attributes of points included in L(N). The update operatormay also smooth the attributes values to improve compression efficiencyof attribute correction values for subsequent levels of detail takinginto account the relative importance of the respective attribute values,wherein the smoothing operation is performed such that attribute valuesthat have a larger impact on subsequent levels of detail are modifiedless than points that have a lesser impact on subsequent levels ofdetail. Several approaches for performing the update operation aredescribed in more detail below. The updated lower resolution sample oflevel of detail L′(N) is then fed to another split operator and theprocess repeats for a subsequent level of detail, LOD(N−1). Note thatattribute signals for the lower level of detail, LOD(N−1) may also bereceived at the second (or subsequent) split operator.

FIG. 13B illustrates an inverse transformation that may be applied at adecoder to decode attribute information of a point cloud, according tosome embodiments.

In some embodiments, a decoder may utilize an inverse transformationprocess as shown in FIG. 13B to reconstruct a point cloud from acompressed point cloud. For example, in some embodiments, performingprediction as described in FIG. 12B at 1258, applying an update operatoras described in FIG. 12B at 1260, applying attribute correction valuesas described in FIG. 12B at 1262 and assigning attributes to points in alevel of detail as described in FIG. 12B at 1264, may be performedaccording to an inverse transformation process as described in FIG. 13B.

In some embodiments, an inverse transformation process may receive anupdated low level resolution sample L′(0) for a lowest level of detailof a LOD structure. The inverse transformation process may also receiveattribute correction values for points not included in the updated lowresolution sample L′(0). For example, for a particular LOD, L′(0) mayinclude a sub-sampling of the points included in the LOD and aprediction technique may be used to determine other points of the LOD,such as would be included in a high resolution sample of the LOD. Theattribute correction values may be received as indicated at 1306, e.g.D(0). At 1318 an update operation may be performed to account for thesmoothing of the attribute correction values performed at the encoder.For example, update operation 1318 may “undo” the update operation thatwas performed at 1314, wherein the update operation performed at 1314was performed to improve compression efficiency by smoothing theattribute values taking into account relative importance of theattribute values. The update operation may be applied to the updated lowresolution sample L′(0) to generate an “un-smoothed” or non-updated lowresolution sample, L(0). The low resolution sample L(0) may be used by aprediction technique at 1320 to determine attribute values of points notincluded in the low resolution sample. The predicted attribute valuesmay be corrected using the attribute correction values, D(0), todetermine attribute values for points of a high resolution sample of theLOD(0). The low resolution sample and the high resolution sample may becombined at merge operator 1322, and a new updated low resolution samplefor a next level of detail L′(1) may be determined. A similar processmay be repeated for the next level of detail LOD(1) as was described forLOD(0). In some embodiments, an encoder as described in FIG. 13A and adecoder as described in FIG. 13B may repeat their respective processesfor N levels of detail of a point cloud.

More detailed example definitions of LODs and methods to determineupdate operations are described below.

In some embodiments, LODs are defined as follows:

-   -   LOD(0)=R(0)    -   LOD(1)=LOD(0) U R(1)    -   . . .    -   LOD(j)=LOD(j−1) U R(j)    -   . . .    -   LOD(N+1)=LOD(N) U R(N)=entire point cloud

In some embodiments, let A be a set of attributes associated with apoint cloud. More precisely, let A(P) be the scalar/vector attributeassociated with the point P of the point cloud. An example of attributewould be color described by RGB values.

Let L(j) be the set of attributes associated with LOD(j) and H(j) thoseassociated with R(j). Based on the definition of level of detailsLOD(j), L(j) and H(j) verify the following properties:

-   -   L(N+1)=A and H(N+1)={ }    -   L(j)=L(j−1) U H(j)    -   L(j) and H(j) are disjointed.

In some embodiments, a split operator, such as split operator 1302,takes as input L(j+1) and generates two outputs: (1) the low resolutionsamples L(j) and (2) the high resolution samples H(j).

In some embodiments, a merge operator, such as merge operator 1322,takes as input L(j) and H(j) and produces L(j+1).

As described in more detail above, a prediction operator may be definedon top of an LOD structure. Let (P(i,j))_i be the set points of LOD(j)and (Q(i,j))_i those belonging to R(j) and let (A(P(i,j)))_i and(A(Q(i,j)))_i be the attribute values associated with LOD(j) and R(j),respectively.

In some embodiments, a prediction operator predicts the attribute valueA(Q(i, j)) by using the attribute values of its k nearest neighbors inLOD(j−1), denoted ∇(Q(i, j)):

${{Pred}\left( {Q\left( {i,j} \right)} \right)} = {\sum\limits_{P \in {\nabla{({Q({i,j})})}}}{{\alpha\left( {P,{Q\left( {i,j} \right)}} \right)}{A(P)}}}$where α(P, Q(i, j)) are the interpolation weights. For instance, aninverse distance weighted interpolation strategy may be exploited tocompute the interpolation weights.

The prediction residuals, e.g. attribute correction values, D(Q(i, j))are defined as follows:D(Q(i,j))=A(Q(i,j))−Pred(Q(i,j))

Note that the prediction hierarchy could be described by an orientedgraph G defined as follows:

-   -   Every point Q in point cloud corresponds to a vertex V(Q) of        graph G.    -   Two vertices of the graph G, V(P) and V(Q), are connected by an        edge E(P, Q), if there exist i and j such that        -   Q=Q(i, j) and        -   P∈∇(Q(i, j))    -   The edge E(Q, P), has weight α(P, Q(i, j)).

In such a prediction strategy as described above, points in lower levelsof detail are more influential since they are used more often forprediction.

Let w(P) be the influence weight associated with a point P. w(P) couldbe defined in various ways.

-   -   Approach 1        -   Two vertices V(P) and V(Q) of G are said to be connected if            there is a path x=(E(1), E(2), . . . , E(s)) of edges of G            that connects them. The weight w(x) of the path x is            defined, as follows:

${w(x)} = {\prod\limits_{s = 1}^{s}{\alpha\left( {E(s)} \right)}}$

-   -   -   Let X(P) be the set of paths having P as destination. w(P)            is defined as follows:            w(P)=1+Σ_(x∈X(P))(w(x))²  [EQ. 1]        -   The previous definition could be interpreted as follows.            Suppose that the attribute A(P) is modified by an amount ∈,            then all the attributes associated with points connected to            P are perturbed. Sum of Squared Errors associated with such            perturbation, denoted SSE(P, ∈) is given by:            SSE(P,∈)=w(P)∈²

    -   Approach 2        -   Computing the influence weights as described previously may            be computationally complex, because all the paths need to be            evaluated. However, since the weights α(E(s)) are usually            normalized to be between 0 and 1, the weight w(x) of a path            x decays rapidly with the number of its edges. Therefore,            long paths could be ignored without significantly impacting            the final influence weight to be computed.        -   Based on the previous property, the definition in [EQ. 1]            may be modified to only consider paths with a limited length            or to discard paths with weights known to be lower that a            user-defined threshold. This threshold could be fixed and            known at both the encoder and decoder, or could be            explicitly signaled at or predefined for different stages of            the encoding process, e.g. once for every frame, LOD, or            even after a certain number of signaled points.

    -   Approach 3        -   w(P) could be approximated by the following recursive            procedure:            -   Set w(P)=1 for all points            -   Traverse the points according to the inverse of the                order defined by the LOD structure            -   For every point Q(i, j), update the weights of its                neighbors P∈∇(Q(i, j)) as follows:                w(P)←w(P)+w(Q(i,j),j){α(P,Q(i,j))}^(γ)                -   where γ is a parameter usually set to 1 or 2.

    -   Approach 4        -   w(P) could be approximated by the following recursive            procedure:            -   Set w(P)=1 for all points            -   Traverse the points according to the inverse of the                order defined by the LOD structure            -   For every point Q(i, j), update the weights of its                neighbors P∈∇(Q(i, j)) as follows:                w(P)←w(P)+w(Q(i,j),j)ƒ{α(P,Q(i,j))}                -   where f(x) is some function with resulting values in                    the range of [0, 1].

In some embodiments, an update operator, such as update operator 1314 or1318, uses the prediction residuals D(Q(i,j)) to update the attributevalues of LOD(j). The update operator could be defined in differentways, such as:

-   -   Approach 1        -   1. Let Δ(P) be the set of points Q(i, j) such that P∈∇(Q(i,            j)).        -   2. The update operation for P is defined as follows:

${{Update}(P)} = \frac{\sum\limits_{Q \in {\Delta(P)}}\left\lbrack {\left\{ {\alpha\left( {P,Q} \right)} \right\}^{\gamma} \times {w(Q)} \times {D(Q)}} \right\rbrack}{\sum\limits_{Q \in {\Delta(P)}}\left\lbrack {\left\{ {\alpha\left( {P,Q} \right)} \right\}^{\gamma} \times {w(Q)}} \right\rbrack}$

-   -   where γ is a parameter usually set to 1 or 2.    -   Approach 2        -   1. Let Δ(P) be the set of points Q(i, j) such that P∈∇(Q(i,            j)).        -   2. The update operation for P is defined as follows:

${{Update}(P)} = \frac{\sum\limits_{Q \in {\Delta(P)}}\left\lbrack {g\left\{ {\alpha\left( {P,Q} \right)} \right\} \times {w(Q)}{D(Q)}} \right\rbrack}{\sum\limits_{Q \in {\Delta(P)}}\left\lbrack {g\left\{ {\alpha\left( {P,Q} \right)} \right\} \times {w(Q)}} \right\rbrack}$

-   -   -   -   where g(x) is some function with resulting values in the                range of [0, 1].

    -   Approach 3        -   Compute Update(P) iteratively as follows:            -   1. Initially set Update(P)=0            -   2. Traverse the points according to the inverse of the                order defined by the LOD structure            -   3. For every point Q(i, j), compute the local updates                (u(1), u(2), . . . , u(k)) associated with its neighbors                ∇(Q(i,j))={P(1), P(2), . . . , P(k)} as the solution to                the following minimization problem:                (u(1),u(2), . . . ,u(k))=argmin{Σ_(r=1)                ^(k)(u(r))²+(D(Q(i,j))−Σ_(r=1) ^(k)α(P(r),Q(i,j))u(k))²}            -   4. Update Update(P(r)):                Update(P(r))←Update(P(r))+u(r)

    -   Approach 4        -   Compute Update(P) iteratively as follows:            -   1. Initially set Update(P)=0            -   2. Traverse the points according to the inverse of the                order defined by the LOD structure            -   3. For every point Q(i, j), compute the local updates                (u(1), u(2), . . . , u(k)) associated with its neighbors                ∇(Q(i, j))={P(1), P(2), . . . , P(k)} as the solution to                the following minimization problem:                (u(1),u(2), . . . ,u(k))=argmin{h(u(1), . . .                ,u(k),D(Q(i,j)))}                -   Where h can be any function.            -   4. Update Update(P(r)):                Update(P(r))←Update(P(r))+u(r)

In some embodiments, when leveraging a lifting scheme as describedabove, a quantization step may be applied to computed waveletcoefficients. Such a process may introduce noise and a quality of areconstructed point cloud may depend on the quantization step chosen.Furthermore, as discussed above, perturbing the attributes of points inlower LODs may have more influence on the quality of the reconstructedpoint cloud than perturbing attributes of points in higher LODs.

In some embodiments, the influence weights computed as described abovemay further be leveraged during the transform process in order to guidethe quantization process. For example, the coefficients associated witha point P may be multiplied with a factor of {w(P)}^(β), where β is aparameter usually set to β=0.5. An inverse scaling process by the samefactor is applied after inverse quantization on the decoder side.

In some embodiments, the values of the β parameters could be fixed forthe entire point cloud and known at both the encoder and decoder, orcould be explicitly signaled at or predefined for different stages ofthe encoding process, e.g. once for every point cloud frame, LOD, oreven after a certain number of signaled points.

In some embodiments, a hardware-friendly implementation of the liftingscheme described above may leverage a fixed-point representation of theweights and lookup tables for the non-linear operations.

In some embodiments, a lifting scheme as described herein may beleveraged for other applications in addition to compression, such asde-noising/filtering, watermarking, segmentation/detection, as well asvarious other applications.

In some embodiments, a decoder may employ a complimentary process asdescribed above to decode a compressed point cloud compressed using anoctree compression technique and binary arithmetic encoder as describedabove.

In some embodiments, the lifting scheme as described above may furtherimplement a bottom-up approach to building the levels of detail (LOD).For example, instead of determining predicted values for points and thenassigning the points to different levels of detail, the predicted valuesmay be determined while determining which points are to be included inwhich level of detail. Also, in some embodiments, residual values may bedetermined by comparing the predicted values to the actual values of theoriginal point could. This too may be performed while determining whichpoints are to be included in which levels of detail. Also, in someembodiments, an approximate nearest neighbor search may be used insteadof an exact nearest neighbor search to accelerate level of detailcreation and prediction calculations. In some embodiments, abinary/arithmetic encoder/decoder may be used to compress/decompressquantized computed wavelet coefficients.

As discussed above, a bottom-up approach may build levels of detail(LODs) and compute predicted attribute values simultaneously. In someembodiments, such an approach may proceed as follows:

-   -   Let (P_(i))_(i=1 . . . N) be the set of positions associated        with the point cloud points and let (M_(i))_(i=1 . . . N) be the        Morton codes associated with (P_(i))_(i=1 . . . N). Let D₀ and ρ        be the two user-defined parameters specifying the initial        sampling distance and the distance ratio between LODs,        respectively. A Morton code may be used to represent        multi-dimensional data in one dimension, wherein a “Z-Order        function” is applied to the multidimensional data to result in        the one dimensional representation. Note that ρ>1    -   First the points are sorted according to their associated Morton        codes in an ascending order. Let I be the array of point indexes        ordered according to this process.    -   The algorithm proceeds iteratively. At each iteration k, the        points belonging to the LOD k are extracted and their predictors        are built starting from k=0 until all the points are assigned to        an LOD.    -   The sampling distance D is initialized with D=D₀    -   For each iteration k, where k=0 . . . Number of LODs        -   Let L(k) be the set of indexes of the points belonging to            k-th LOD and O(k) the set of points belonging to LODs higher            than k. L(k) and O(k) are computed as follows.        -   First, O(k) and L(k) are initialized            -   if k=0, L(k)←{ }. Otherwise, L(k)←L(k−1)            -   O(k)←{ }        -   The point indexes stored in the array I are traversed in            order. Each time an index i is selected and its distance            (e.g., Euclidean or other distance) to the most recent SR1            points added to O(k) is computed. SR1 is a user-defined            parameter that controls the accuracy of the nearest neighbor            search. For instance, SR1 could be chosen as 8 or 16 or 64,            etc. The smaller the value of SR1 the lower the            computational complexity and the accuracy of the nearest            neighbor search. The parameter SR1 is included in the bit            stream. If any of the SR1 distances is lower than D, then i            is appended to the array L(k). Otherwise, i is appended to            the array O(k).            -   The parameter SR1 could be changed adaptively based on                the LOD or/and the number of points traversed.            -   In some embodiments, instead of computing an approximate                nearest neighbor, an exact nearest neighbor search                technique may be applied.            -   In some embodiments, the exact and approximate neighbor                search methods could be combined. In particular,                depending on the LOD and/or the number of points in I,                the method could switch between the exact and                approximate search method. Other criteria, may include                the point cloud density, the distance between the                current point and the previous one, or any other                criteria related to the point cloud distribution.        -   This process is iterated until all the indexes in I are            traversed.        -   At this stage, L(k) and O(k) are computed and will be used            in the next steps to build the predictors associated with            the points of L(k).        -   More precisely, let R(k)=L(k)\L(k−1) (where \ is the            difference operator) be the set of points that need to be            added to LOD(k−1) to get LOD(k). For each point i in R(k),            we would like to find the h-nearest neighbors (h is            user-defined parameters that controls the maximum number of            neighbors used for prediction) of i in O(k) and compute the            prediction weights (α_(j)(i))_(j=1 . . . h) associated            with i. The algorithm proceeds as follows.        -   Initialize a counter j=0        -   For each point i in R(k)            -   Let M_(i) be the Morton code associated with i and let                M_(j) be the Morton code associated with j-th element of                the array O(k)            -   While (M_(i)≥M_(j) and j<SizeOf(O(k))), incrementing the                counter j by one (j←j+1)            -   Compute the distances of M_(i) to the points associated                with the indexes of O(k) that are in the range [j−SR2,                j+SR2] of the array and keep track of the h-nearest                neighbors (n₁, n₂, . . . , n_(h)) and their associated                squared distances(d_(n) ₁ ²(i), d_(n) ₂ ²(i) . . . ,                d_(n) _(h) ²(i)). SR2 is a user-defined parameter that                controls the accuracy of the nearest neighbor search.                Possible values for SR2 are 8, 16, 32, and 64. The                smaller the value of SR2 the lower the computational                complexity and the accuracy of the nearest neighbor                search. The parameter SR2 is included in the bit stream.                The computation of the prediction weights used for                attribute prediction may be the same as described above.                -   The parameter SR2 could be changed adaptively based                    on the LOD or/and the number of points traversed.                -   In some embodiments, instead of computing an                    approximate nearest neighbor, an exact nearest                    neighbor search technique may be used.                -   In some embodiments, the exact and approximate                    neighbor search methods could be combined. In                    particular, depending on the LOD and/or the number                    of points in I, the method could switch between the                    exact and approximate search method. Other criteria,                    may include the point cloud density, the distance                    between the current point and the previous one, or                    any other criteria related to the point cloud                    distribution.                -   If the distance between the current point and the                    last processed point is lower than a threshold, use                    the neighbors of the last point as an initial guess                    and search around them. The threshold could be                    adaptively chosen based on similar criteria as those                    described above. The threshold could be signaled in                    the bit stream or known to both encoder and decoder.                -   The previous idea could be generalized to n=1, 2, 3,                    4 . . . last points                -   Exclude points with a distance higher that a                    user-defined threshold. The threshold could be                    adaptively chosen based on similar criteria as those                    described above. The threshold could be signaled in                    the bitstream or known to both encoder and decoder.        -   I←O(k)        -   D←D×ρ        -   The approach described above could be used with any metric            (e.g., L2, L1, Lp) or any approximation of these metrics.            For example, in some embodiments distance comparisons may            use a Euclidean distance comparison approximation, such as a            Taxicab/Manhattan/L1 approximation, or an Octagonal            approximation.

In some embodiments, a lifting scheme may be applied without determininga hierarchy of levels. In such embodiments, the technique may proceed asfollows:

-   -   Sort the input points according to the Morton codes associated        with their coordinates    -   Encode/decode point attributes according to the Morton order    -   For each point i, look for the h nearest neighbors (n₁, n₂, . .        . , n_(h)) already processed (n_(j)<i)    -   Compute the prediction weights as described above.    -   Apply the adaptive scheme described above in order to adjust the        prediction strategy.    -   Predict attributes and entropy encode them as described below.        Binary Arithmetic Coding of Quantized Lifting Coefficients

In some embodiments, lifting scheme coefficients may be non-binaryvalues. In some embodiments, an arithmetic encoder, such as arithmeticencoder 212, that was described above as a component of encoded 202illustrated in FIG. 2B and which used a binary arithmetic codec toencode the 256-value occupancy symbols may also be used to encodelifting scheme coefficients. Or, in some embodiments a similararithmetic encoder may be used. For example, the technique may proceedas follows:

-   -   Mono-dimensional attribute        -   Let C be the quantized coefficient to be encoded. First C is            mapped to a positive number using a function that maps            positive numbers to even numbers and negative numbers to odd            numbers.        -   Let M(C) be the mapped value.        -   A binary value is then encoded to indicate whether C is 0 or            not        -   If C is not zero, then two cases are distinguished            -   If M(C) is higher or equal than alphabetSize (e.g. the                number of symbols supported by the binary arithmetic                encoding technique described above in regard to FIGS.                12C-12F), then the value alphabetSize is encoded by                using the method described above. The difference between                M(C) and alphabetSize is encoded by using an exponential                Golomb coding            -   Otherwise, the value of M(C) is encoded using the method                described above in regard to FIGS. 12C-12F for N.    -   Three-dimensional signal        -   Let C1, C2, C3 be the quantized coefficients to be encoded.            Let K1, and K2 be two indexes for the contexts to be used to            encode the quantized coefficients, C1, C2, and C3.        -   First C1, C2 and C3 are mapped to a positive number as            described above in regard to FIGS. 12C-12F. Let M(C1), M(C2)            and M(C3) be the mapped values of C1, C2, and C3.        -   M(C1) is encoded.        -   M(C2) is encoded while choosing different contexts (i.e.,            binary arithmetic contexts and the binarization context            defined above in regards to FIG. 12C-12F) based on the            condition of whether C1 is zero or not.        -   M(C3) is encoded while choosing different contexts based on            the conditions C1 is zero or not and C2 is zero or not. If            C1 is zero it is known that the value is at least 16. If the            condition C1 is zero use the binary context K1, if the value            is not zero, decrement the value by 1 (it is known that the            value is at least one or more), then check the value is            below the alphabet size, if so encode the value directly.            Otherwise, encode maximum possible value for the alphabet            size. The difference between the maximum possible value for            the alphabet size and the value of M(C3) will then be            encoded using exponential Golomb encoding.    -   Multi-dimensional signal        -   The same approach described above could be generalized to a            d-dimensional signal. Here, the contexts to encode the k-th            coefficient depending on the values of the previous            coefficients (e.g., last 0, 1, 2, 3, . . . , k−1            coefficients).        -   The number of previous coefficients to consider could be            adaptively chosen depending on any of the criteria described            in the previous section for the selection of SR1 and SR2.

Below is a more detailed discussion of how a point cloud transferalgorithm is utilized to minimize distortion between an original pointcloud and a reconstructed point cloud.

The attribute transfer problem could be defined as follows:

-   -   a. Let PC1=(P1(i))_(i∈{1, . . . , N1}) be a point cloud defined        by its geometry (i.e., 3D positions) (X1(i))_(i∈{1, . . . , N1})        and a set of attributes (e.g., RGB color or reflectance)        (A(i))_(i∈{1, . . . , N1}).    -   b. Let PC2(P2(j))_(j∈{1, . . . , N2}) be a re-sampled version of        PC1 and let (X2(j))_(j∈{1, . . . , N2}) be its geometry.    -   c. Then compute the set of attribute of        (A2(j))_(j∈{1, . . . , N2}) associated with the point of PC2        such that the texture distortion is minimized.

In order to solve the texture distortion minimization problem using anattribute transfer algorithm:

-   -   Let P_(2→1)(j)∈PC1 be the nearest neighbor of P2(j)∈PC2 in PC1        and A_(2→1)(j) its attribute value.    -   Let P_(1→2)(i)∈PC2 be the nearest neighbor of P1(i)∈PC1 in PC2        and A_(1→2)(i) its attribute value.    -   Let        _(1→2)(j)=(Q(j, h))_(h∈{1, . . . , H(j)}) ⊆PC2 be the set of        points of PC2 that share the point P1(i)∈PC1 as their nearest        neighbor and (α(j, h))_(h∈{1, . . . , H(j)}) be their attribute        values    -   Let E_(2→1) be the non-symmetric error computed from PC2 to PC1:        E _(2→1)=Σ_(j=1) ^(N2) ∥A2(j)−A _(2→1)(j)∥²    -   Let E_(1→2) be the non-symmetric error computed from PC1 to PC2:        E _(1→2)=Σ_(i=1) ^(N1) ∥A1(j)−A _(1→2)(j)∥²    -   Let E be symmetric error that measures the attribute distortion        between PC2 to PC1:        E=max(E _(2→1) ,E _(1→2))

Then determine the set of attributes (A2(j))_(j∈{1, . . . , N2}) asfollows:

  a. Initialize E1 ← 0 and E2 ← 0 b. Loop over all the point of PC2   1)For each point P2(j) compute P_(2→1)(j) ∈ PC1 and

_(1→2)(j)   2) If (E1 > E2 or

_(1→2)(j) = { })    • A2(j) = A_(2→1)(j)   3) Else    ${{{A2}(j)}} = {\frac{1}{H(j)}{\sum_{h = 1}^{H(j)}{\alpha\left( {j,h} \right)}}}$  4) EndIf   5) E1 ← E1 + ||A2(j) − A_(2→1)(j)||₂   $\left. {\left. 6 \right){E2}}\leftarrow{{E2} + {{{{A2}(j)} - {\frac{1}{H(j)}{\sum_{h = 1}^{H(j)}{\alpha\left( {j,h} \right)}}}}}^{2}} \right.$Point Cloud Attribute Transfer Algorithm

In some embodiments, a point cloud transfer algorithm may be used tominimize distortion between an original point cloud and a reconstructedversion of the original point cloud. A transfer algorithm may be used toevaluate distortion due to the original point cloud and thereconstructed point cloud having points that are in slightly differentpositions. For example, a reconstructed point cloud may have a similarshape as an original point cloud, but may have a.) a different number oftotal points and/or b.) points that are slightly shifted as compared toa corresponding point in the original point cloud. In some embodiments,a point cloud transfer algorithm may allow the attribute values for areconstructed point cloud to be selected such that distortion betweenthe original point cloud and a reconstructed version of the originalpoint cloud is minimized. For example, for an original point cloud, boththe positions of the points and the attribute values of the points areknown. However, for a reconstructed point cloud, the position values maybe known (for example based on a sub-sampling process, K-D tree process,or patch image process as described above). However, attribute valuesfor the reconstructed point cloud may still need to be determined.Accordingly a point cloud transfer algorithm can be used to minimizedistortion by selecting attribute values for the reconstructed pointcloud that minimize distortion.

The distortion from the original point cloud to the reconstructed pointcloud can be determined for a selected attribute value. Likewise thedistortion from the reconstructed point cloud to the original pointcloud can be determined for the selected attribute value for thereconstructed point cloud. In many circumstances, these distortions arenot symmetric. The point cloud transfer algorithm is initialized withtwo errors (E21) and (E12), where E21 is the error from the second orreconstructed point cloud to the original or first point cloud and E12is the error from the first or original point cloud to the second orreconstructed point cloud. For each point in the second point cloud, itis determined whether the point should be assigned the attribute valueof the corresponding point in the original point cloud, or an averageattribute value of the nearest neighbors to the corresponding point inthe original point cloud. The attribute value is selected based on thesmallest error.

Trimming Search Space for Nearest Neighbor Searches for Point CloudCompression and Decompression

In various embodiments, techniques for generating levels of detail(LODs) as discussed above may iteratively apply a subsampling process inorder to separate points that are in a current LOD from those belongingto the next LODs. In various embodiments, the process may includechecking a distance for each point in terms of a Euclidean norm(sometimes referred to as an “L2” norm) for points represented asvectors to all other points in a point cloud that occur before thecurrent point in an order for generating LODs (e.g., a Morton order). Ifany of the distances is higher than a defined threshold, then the pointmay be included in the current LOD. Otherwise, the point may belong tothe subsequent LODs. By trimming the search to a limited search range,as discussed below in various embodiments (e.g., a search range 1 “SR1”of 64 or 128), an approximate nearest neighbor search can be performedthat is an order of magnitude faster than other nearest neighbor searchtechniques which do not utilize approximation.

For example, in some embodiments, the search space for nearest neighborsearch may be trimmed (or otherwise reduced) by implementing boundingshapes. Consider an example embodiment where the points of a point cloudare clustered, grouped, or otherwise assigned into point buckets of afixed size, as illustrated in FIG. 14 . Each bucket, such as buckets1420 a, 1420 b, 1420 c, and so on, may be assigned a range of spacefilling curve values 1400 determined points in a point cloud, such asspace filling curve ranges 1410 a, 1410 b, and 1410 c respectivelyassigned to buckets 1420 a, 1420 b, and 1420 c. Consider an exampleembodiment where each 8, 16 or 32 consecutive points in a Morton ordercould be grouped in a point bucket 1420. Bounding shapes, such as anaxis aligned bounding box 1430 may then be determined and associatedwith each bucket (as bounding box 1430 may be associated with the pointsincluded in bucket 1420 a. In various embodiments, bounding boxes couldbe computed incrementally or pre-computed before the starting thesub-sampling process. Although a bounding box is illustrated anddiscussed with regard to FIG. 14 , other bounding shapes, such asbounding spheres, bounding capsules, and so on, may be implemented inother embodiments.

Bounding shapes may be used to compute a minimal bound on the distancefrom a point outside of a point bucket to a point assigned to the pointbucket. For example, as indicated in FIG. 14 , a minimum distance topoints 1440 may be determined with respect to bounding box 1430 (insteadof computing a distance between an external point and a point within thebucket 1420 a). FIG. 15 illustrates a high-level flowchart for applyingbounding shapes to trim search space for nearest neighbor searching,according to some embodiments.

As indicated at 1510, points of a point cloud may be grouped withindifferent ranges of a space filling curve that include respective spacefilling curve values generated for the points, in some embodiments. Forexample, a Morton code may be generated for points of a point cloud.Points with Morton code values that fall within a range of Morton codevalues assigned to group A (e.g., Bucket A) may be grouped in group A.Similar groupings may be made for points within Morton code values thatfall within a range Morton code values assigned to group B, and so on.

As indicated at 1520, bounding shapes may be determined for the groupedpoints within the different ranges of the space filling curve, in someembodiments. For example, a shape that encompasses all points within thegroup, a box, a sphere, a cube, and so on, may be determined. In someembodiments, techniques may be implemented to determine a best fitbounding shape (e.g., which shape encompasses all points but covers thesmallest amount of space). Bounding shapes may be determined as apreprocessing step before subsampling or may be performed iteratively,in some embodiments.

As indicated at 1530, a nearest neighbor search for encoding the pointcloud may be performed, in various embodiments. For example, asdiscussed above with regard to FIGS. 1-13B, various nearest neighborsearches may be performed. To determine whether a group of points shouldbe included in a nearest neighbor search, a distance between a point forwhich the nearest neighbor search is performed and the bounding shapesof the group may be determined, as indicated at 1540. A comparison ofthe determined distances and a sampling threshold may be made. If thesampling threshold is exceeded, then as indicated at 1550, those pointsin the groups with distances to the respective bounding shapes thatexceeded the threshold may be excluded from the nearest neighbor search.In this way, the search space may be reduced, by not individualcomputing distances for points within an excluded bounding shape. Forgroups within bounding shapes that do not exceed the threshold,individual distance determinations may be made with the points in thegroup and the point being searched. Those distances within the samplingthreshold may be included, in some embodiments.

In some embodiments, the techniques discussed above with regard to FIGS.15 and 16 may include further optimizations. For example, in someembodiments, the distance measure may be changed from an L2 distance(which may be implemented using 3 multiplications operations, whichcould expensive if a HW implementation is considered) and an L1normalization (e.g., which takes the absolute value of k-dimensionalpoints |x|+|y|+|z|, and so on up to k dimensions and which can beimplemented using adders, providing a cheaper hardware implementation),could be used instead. In some embodiments, L1 may be implemented for aninitial distance search and keep a list of potential nearest neighborcandidates. Then, a different distance measure, such as an L2normalization (e.g. which takes the sum of the squares of thek-dimensional points x²+y²+z² and so on up to k dimensions, which mayalso optionally take the square root of the sum of the squares) basedsearch may be applied to refine the nearest neighbor candidates. Notethat in some embodiments, a point cloud may be define in more than threedimensions (e.g. X, Y, and Z), for example in some embodiments, a fourthdimension may be time, etc.

In at least some embodiments, the techniques discussed above forsubsampling to find nearest neighbors for search could be parallelizedby computing the distances in parallel for each point. In someembodiments, the search range could also be trimmed based the twofollowing properties of Morton ordered points, such as by utilizing asmallest quadtree box containing two points that will also contain allpoints lying between the two points in Morton order to determine whetherpoints should be considered for nearest neighbors.

In various embodiments, search space for nearest neighbor searches maybe trimmed for predictor generation techniques. For example, aprediction generation technique may attempt to compute for each point ina current LOD the point's k-nearest neighbors in the subsequent LODs,and/or in the same LOD with lower decoder order. In these and otherpredictor generation scenarios, techniques to trim the search space maybe implemented to improve search performance.

For example, in some embodiments, because nearest neighbor searches maybe applied to all the points in the current LOD, the results of one ormultiple points could be re-used for searching other points. In someembodiments, a nearest neighbor search may be performed within a searchrange (SR2) for even points in a space filling curve, such as Mortonorder. For odd points in the space filling curve, the results from theevent point distance calculations could be reused, and thus reducing thesearch space. In various embodiments, the choice of which points wouldbenefit from a nearest neighbor search with a search range SR2 and whichones would re-use search results could be adaptive based on inter-pointdistance criteria. Some points could also be combined re-using otherpoints searches with a limited local search with a search range SR3lower than SR2.

FIG. 16 illustrates an example of nearest neighbor search result reuseaccording to ranges of a space filling curve, according to someembodiments. Space filling curve 1600, which may be a Morton order orother space filling curve, may illustrated different space filling curvevalue ranges, such as ranges 1610 a, 1610 b, 1610 c, 1610 d, 16010 e,and so on. As discussed in the example above, the ranges could bealternating odd and even space filling curve values. However, in otherembodiments different ranges (including differently sized ranges) may beimplemented. Different ranges of the space filling curve may beidentified for new search results, generated for a current LOD and priorsearch results generated for a prior LOD. For example, for a nearestneighbor search 1640, ranges 1610 a, 1610 c and 1610 e may be identifiedfor reuse, so that search results (e.g., distance values that determinewhich points should be sampled as a nearest neighbor or not) can bereused from a prior LOD nearest neighbor search, as indicated at 1620 a,1620 c and 1620 e. Some search ranges may be identified for generatingnew results, such as space filling curve ranges 1610 b and 1610 d, whichuse search results from the current LOD 1620 b and 1620 d respectively.

FIG. 17 illustrates a high-level flowchart for applying bounding shapesto trim search space for nearest neighbor searching, according to someembodiments. As indicated at 1710, nearest neighbor search results forpoints of a first level of detail (LOD) may be generated as part ofsubsampling to generate predictors, according to various techniquesdiscussed above. As indicated at 1720, nearest neighbor search resultsfor points of a second LOD of the point cloud may be generated accordingwith space filling curve values that are within ranges identified fornew search results, according to some embodiments. As indicated at 1730,those portions of the nearest neighbor search results for points of thefirst LOD of the point cloud may be selected for points in the secondLOD with space filling curve values that are within ranges identifiedfor result reuse, according to some embodiments.

Various further optimizations may be implemented in some embodiments.For example, as discussed above different normalization techniques maybe used (e.g., L1 norm instead of L2). In some embodiments, differentnormalizing techniques may be performed in order to filter searchresults, such as using L1 normalization for an initial search and keep alist of potential nearest neighbor candidates. Then, the nearestneighbor candidates list can be refined with an L2-based search. In someembodiments, for nearest neighbor searches in subsequent LODs, thebounding boxes computed in the previous section could be reused to trimthe search space in the same manner as in the previous section,according to the techniques discussed above. In some embodiments, forpoints in the same LOD, new bounding boxes may be computed as describedand used in addition to generate search results for those portionsidentified for new results.

In some embodiments, a k-nearest neighbor graph G could be pre-computedfor the points belonging to subsequent LODs. This graph could be thenused to accelerate the search process. For example, for each point P, aquick search with a limited search range would be applied first to findan approximate nearest neighbor n0. The neighbors of the node in graph Gare then evaluated to find the k-nearest neighbors of P in graph G. Theprocess to generate the graph G could then accelerated by looking fornearest neighbors that have a lower space filling curve value (e.g.,lower Morton order) only and symmetrizing the graph G (e.g., if P is aneighbor of Q then make Q a neighbor of P). In order, to reduce thememory requirements for G, the number of nearest neighbor searches ofgraph G could be adaptively varied based on the point cloudcharacteristics.

All the approaches described above could be combined in differentmanners to achieve different optimizations for reducing the searchspace. As noted earlier, the nearest neighbor search could beparallelized along different dimensions, such as by performing nearestneighbor search in parallel for different points and/or neighbors couldbe searched in parallel for each point, in some embodiments.

Further Enhancements to the Trimmed Search Space for Nearest NeighborSearches

The “K” nearest neighbor problem (k-NN) in the context of both theLifting/Prediction scheme and the region adaptive hierarchical transform(RAHT) scheme, could be formulated as follows and described in terms ofFIG. 18 . Let A and B be two set of points in a discrete metric spaceR{circumflex over ( )}d, where d is the dimension of the space (e.g.,d=3). For each point in A, it is desired to find its k nearest neighborsin B.

As discussed above, both A and B may be sorted according to their Mortonorder. Also as discussed above, an approximate “K” nearest neighborsolution may be computed as follows:

-   -   Let j be a counter initialized to 0    -   Let P(i) be the i-th point of A according to Morton order    -   Keep incrementing j until the Morton code of P(i), denoted        MP(i), satisfies the following condition        -   where MQ(j) is the Morton code of the j-th point of B,            denoted Q(j), according to Morton order.    -   Apply a limited search in the following subset S(j) of points of        B        S(j)={Q(j),Q(j+1),Q(j−1), . . . ,Q(j+Δ),Q(j−Δ)}

Note that when A=B, we can apply the same algorithm with j=i.

Also note that the problem could be further restricted by allowing only“K” nearest neighbors with a lower Morton code. Here, S(j) is defined asfollows:S(j)={Q(j),Q(j−1), . . . ,Q(j−Δ)}

This approach provides a good approximation of “K” nearest-neighbors.However, it may fail to capture the actual nearest neighbors whensignificant jumps in terms of Morton order is observed betweenneighboring points (see Points P and Q in FIG. 19 ).

Because the determined “K” nearest neighbors are used to predictattribute of the points and/or to determine levels of detail (LOD),improving the accuracy of the “K” nearest neighbor search may improvecompression efficiency. For example, a prediction based on more accuratenearest neighbors may yield better prediction results than predictionbased on a set of points that inadvertently excludes one or more pointsthat are closer to the point being evaluated than the points included inthe set of nearest neighboring points, e.g. an excluded nearestneighboring point. Furthermore, if the prediction is more accurate, thenthe residual values, e.g. attribute correction values may also besmaller. Because attribute correction values are encoded in a compressedattribute file, reducing the size of the attribute correction valueswill also reduce the amount of data that has to be encoded andcommunicated to a decoder, thus improving compression efficiency.Additionally, if quantization is applied to the attribute correctionvalues, larger quantized attribute correction values may also impactreconstructed quality of the attributes of the point cloud. Thus,improving prediction accuracy and therefore reducing the size of theattribute correction values by using a more accurate set of neighboringpoints for prediction, may also improve the quality of the reconstructedpoint cloud.

In some embodiments, in order to further refine the “K” nearest neighborsearch, a solution may be computed as follows:

-   -   Let j be a counter initialized to 0    -   Let P(i) be the i-th point of A according to Morton order    -   Keep incrementing j until the Morton code of P(i), denoted        MP(i), satisfies the following condition        MP(i)≥MQ(j),    -    where MQ(j) is the Morton code of the j-th point of B, denoted        Q(j), according to Morton order.    -   Apply a limited search in the following subset S(j) of points of        B        S(j)={Q(j),Q(j+1),Q(j−1), . . . ,Q(j+Δ),Q(j−Δ)}    -   Let N(i, 1), N(i, 2), . . . , N(i, H) be the set of neighbors of        P(i) in R^(d). For instance the 6/18/26-connectivity of the P(i)        union the {P(i)}    -   For each point N(i, h) search if it exists in B        -   Alternative 1: Apply binary search            -   Let MN(i, h) be the Morton code of N(i, h)            -   If MN(i, h)≥MQ(j−Δ) and MN(i, h)≤MQ(j+Δ), do nothing                (already captured if it belongs to B, by the initial                search)            -   If MN(i, h)<MQ(j−Δ), then apply binary search to the                integer interval [j−Δ−1−MQ(j−Δ)+MN(i, h), j−Δ−1]            -   If MN(i, h)>MQ(j+Δ), then apply binary search to the                integer interval [j+Δ+1, j+Δ+1+MQ(j+Δ)−MN(i, h),]        -   Alternative 2: Use look-up table            -   Let C=[0, . . . , 2^(c)−1]×[0, . . . , 2^(c)−1]×[0, . .                . , 2^(c)−1] be the bounding cube of B (i.e., B⊂C)            -   First, a LUT that maps any point X of C to                -   True (or −1) if the point does not belong to B                -   false (or the index of X in B), otherwise.            -   Use the LUT to check if MN(i, h) belongs to B. If                MN(i, h) belongs B, add MN(i, h) to S(j)            -   Note 1: the LUT could be implemented as a table or using                any hierarchical structure (e.g., octree, kd-tree,                hierarchical grids . . . ) to save memory by leveraging                the sparse nature of the point cloud            -   Note 2: The LUT could store multiple indexes per                location, in order to handle duplicated points.            -   Note 3: Allocating a LUT that holds C may be expensive                in terms of memory. To reduce such requirements, the                bounding cube C may be partitioned into smaller                sub-cubes {E(a, b, c)} of size 2^(e). Since the points                are traversed in Morton order, all the points in one                sub-cubes, will be traversed in sequence before                switching the next subcube. Therefore, only a LUT able                to store a single sub-cube is needed. The LUT would be                initialized every time P(i) enters a new sub-cube. Only                the point of B in that sub-cube would be added to the                LUT. For points on the boundary of the sub-cube two                strategies are possible:                -   Ignore neighborhood relationship across sub-cubes                    boundaries                -   Use a binary search (i.e., alternative 1) to                    determine if their neighbors exists or not.        -   Alternative 3: combine alternative 1 and alternative 2

Note, that the connectivity of the neighbors, e.g. a connectivity of 6neighboring voxels may be similar to the connectivity 1286 shown in FIG.12C. While not shown, similar connectivities for more voxels, such as 18or 26 may be used. Also, as shown in FIG. 12B, a cube may be divided andfurther divided into sub-cubes. In some embodiments, the voxel in whichthe point for which the nearest neighbor search is being conducted mayadditional be searched to see if there is another neighboring point inthe same voxel. In that case, the connectivity may be 7, 19, 27, etc.when the voxel in which the point being evaluated resides is furtherincluded in the set of voxels to be considered.

Also, as can be seen above, the binary search to determine if a Mortoncode of a point of the point cloud matches a Morton code of theneighboring voxel, may exclude the Morton code used in the initialsearch based on Morton codes. This is because the points with thoseMorton codes would have already been identified by the first search,thus it is unnecessary to additional search for them a second time inneighboring voxels, if they have already been identified as one of thenearest neighboring points.

Additionally, in some embodiments, in order to further refine the “K”nearest neighbor search, a solution may be computed as follows:

-   -   The search could proceed in the opposite way as just described        above. For example, a search based on the neighboring voxels        (e.g. neighbors N(i,1),N(i,2), . . . , N(i,H) is applied first.        Then the search is further refined by applying the trimmed        Morton order search.    -   Apply a k-NN search based N(i, 1), N(i, 2), . . . , N(i, H)        -   Found no neighbors            -   Search in S(j)={Q(j), Q(j+1), Q(j−1), . . . , Q(j+Δ),                Q(j−Δ)}        -   Found one neighbor            -   Let j* be the index of the found nearest neighbor in B            -   Search in S(j*)={Q(j*), Q(j*+1), Q(j*−1), . . . ,                Q(j*+Δ), Q(j*−Δ)}        -   Found k′<k neighbors. Let j*₁, j*₂, . . . , j*_(k) be the            indexes of the found nearest neighbor in B            -   Approach 1                -   Search in S(j*₁)={Q(j*₁), Q(j*₁+1), Q(j*₁−1), . . .                    , Q(j*₁+Δ), Q(j*₁−Δ)}            -   Approach 2                -   Let be the index of the found nearest neighbor in B                -   Search in S(j*₁)∪S(j*₂)∪ . . . S(j*_(k))

Additionally, in some embodiments, a lifting scheme, such as describedin FIGS. 13A-13B, and/or a regional adaptive hierarchical transform(RAHT) scheme, may further be leveraged when searching for the “K”nearest neighbors. For example, the lifting/prediction scheme guaranteesthat at each LOD l the distance between each two points is higher than apre-defined threshold d(l). The RAHT scheme merges at each level all thepoints sharing the same Morton codes shifted by l bits. If the sequencesof distances is constrained {d(1), d(2), . . . d(L)}, as follows, thesame properties as the RAHT may be achieved:d(1)=√{square root over (3)}×2^(n0)d(l+1)=2×d(l)

The example described below the focus is on the Lifting/PredictionScheme. However, a similar approach may also be applied to the RAHTscheme.

Based on the constrained distance model described above, it is knownthat all points of LOD l=2 have a distance of higher than√3×2{circumflex over ( )}(n0+1). Therefore, if the coordinates of thepoints of B by 2{circumflex over ( )}(n0+1) are divided (e.g., shift allthe coordinates by (n0+1) bits) all the points would still have distinctcoordinates, while significantly shrinking the size of the LUTs and/orthe range of binary searches described in the previous section.

Example Applications for Point Cloud Compression and Decompression

FIG. 20 illustrates compressed point clouds being used in a 3-Dtelepresence application, according to some embodiments.

In some embodiments, a sensor, such as sensor 102, an encoder, such asencoder 104 or encoder 202, and a decoder, such as decoder 116 ordecoder 220, may be used to communicate point clouds in a 3-Dtelepresence application. For example, a sensor, such as sensor 102, at2002 may capture a 3D image and at 2004, the sensor or a processorassociated with the sensor may perform a 3D reconstruction based onsensed data to generate a point cloud.

At 2006, an encoder such as encoder 104 or 202 may compress the pointcloud and at 2008 the encoder or a post processor may packetize andtransmit the compressed point cloud, via a network 2010. At 2012, thepackets may be received at a destination location that includes adecoder, such as decoder 116 or decoder 220. The decoder may decompressthe point cloud at 2014 and the decompressed point cloud may be renderedat 2016. In some embodiments a 3-D telepresence application may transmitpoint cloud data in real time such that a display at 2016 representsimages being observed at 2002. For example, a camera in a canyon mayallow a remote user to experience walking through a virtual canyon at2016.

FIG. 21 illustrates compressed point clouds being used in a virtualreality (VR) or augmented reality (AR) application, according to someembodiments.

In some embodiments, point clouds may be generated in software (forexample as opposed to being captured by a sensor). For example, at 2102virtual reality or augmented reality content is produced. The virtualreality or augmented reality content may include point cloud data andnon-point cloud data. For example, a non-point cloud character maytraverse a landscape represented by point clouds, as one example. At2104, the point cloud data may be compressed and at 2106 the compressedpoint cloud data and non-point cloud data may be packetized andtransmitted via a network 2108. For example, the virtual reality oraugmented reality content produced at 2102 may be produced at a remoteserver and communicated to a VR or AR content consumer via network 2108.At 2110, the packets may be received and synchronized at the VR or ARconsumer's device. A decoder operating at the VR or AR consumer's devicemay decompress the compressed point cloud at 2112 and the point cloudand non-point cloud data may be rendered in real time, for example in ahead mounted display of the VR or AR consumer's device. In someembodiments, point cloud data may be generated, compressed,decompressed, and rendered responsive to the VR or AR consumermanipulating the head mounted display to look in different directions.

In some embodiments, point cloud compression as described herein may beused in various other applications, such as geographic informationsystems, sports replay broadcasting, museum displays, autonomousnavigation, etc.

Example Computer System

FIG. 22 illustrates an example computer system 2200 that may implementan encoder or decoder or any other ones of the components describedherein, (e.g., any of the components described above with reference toFIGS. 1-21 ), in accordance with some embodiments. The computer system2200 may be configured to execute any or all of the embodimentsdescribed above. In different embodiments, computer system 2200 may beany of various types of devices, including, but not limited to, apersonal computer system, desktop computer, laptop, notebook, tablet,slate, pad, or netbook computer, mainframe computer system, handheldcomputer, workstation, network computer, a camera, a set top box, amobile device, a consumer device, video game console, handheld videogame device, application server, storage device, a television, a videorecording device, a peripheral device such as a switch, modem, router,or in general any type of computing or electronic device.

Various embodiments of a point cloud encoder or decoder, as describedherein may be executed in one or more computer systems 2200, which mayinteract with various other devices. Note that any component, action, orfunctionality described above with respect to FIGS. 1-21 may beimplemented on one or more computers configured as computer system 2200of FIG. 22 , according to various embodiments. In the illustratedembodiment, computer system 2200 includes one or more processors 2210coupled to a system memory 2220 via an input/output (I/O) interface2230. Computer system 2200 further includes a network interface 2240coupled to I/O interface 2230, and one or more input/output devices2250, such as cursor control device 2260, keyboard 2270, and display(s)2280. In some cases, it is contemplated that embodiments may beimplemented using a single instance of computer system 2200, while inother embodiments multiple such systems, or multiple nodes making upcomputer system 2200, may be configured to host different portions orinstances of embodiments. For example, in one embodiment some elementsmay be implemented via one or more nodes of computer system 2200 thatare distinct from those nodes implementing other elements.

In various embodiments, computer system 2200 may be a uniprocessorsystem including one processor 2210, or a multiprocessor systemincluding several processors 2210 (e.g., two, four, eight, or anothersuitable number). Processors 2210 may be any suitable processor capableof executing instructions. For example, in various embodimentsprocessors 2210 may be general-purpose or embedded processorsimplementing any of a variety of instruction set architectures (ISAs),such as the x86, PowerPC, SPARC, or MIPS ISAs, or any other suitableISA. In multiprocessor systems, each of processors 2210 may commonly,but not necessarily, implement the same ISA.

System memory 2220 may be configured to store point cloud compression orpoint cloud decompression program instructions 2222 and/or sensor dataaccessible by processor 2210. In various embodiments, system memory 2220may be implemented using any suitable memory technology, such as staticrandom access memory (SRAM), synchronous dynamic RAM (SDRAM),nonvolatile/Flash-type memory, or any other type of memory. In theillustrated embodiment, program instructions 2222 may be configured toimplement an image sensor control application incorporating any of thefunctionality described above. In some embodiments, program instructionsand/or data may be received, sent or stored upon different types ofcomputer-accessible media or on similar media separate from systemmemory 2220 or computer system 2200. While computer system 2200 isdescribed as implementing the functionality of functional blocks ofprevious Figures, any of the functionality described herein may beimplemented via such a computer system.

In one embodiment, I/O interface 2230 may be configured to coordinateI/O traffic between processor 2210, system memory 2220, and anyperipheral devices in the device, including network interface 2240 orother peripheral interfaces, such as input/output devices 2250. In someembodiments, I/O interface 2230 may perform any necessary protocol,timing or other data transformations to convert data signals from onecomponent (e.g., system memory 2220) into a format suitable for use byanother component (e.g., processor 2210). In some embodiments, I/Ointerface 2230 may include support for devices attached through varioustypes of peripheral buses, such as a variant of the Peripheral ComponentInterconnect (PCI) bus standard or the Universal Serial Bus (USB)standard, for example. In some embodiments, the function of I/Ointerface 2230 may be split into two or more separate components, suchas a north bridge and a south bridge, for example. Also, in someembodiments some or all of the functionality of I/O interface 2230, suchas an interface to system memory 2220, may be incorporated directly intoprocessor 2210.

Network interface 2240 may be configured to allow data to be exchangedbetween computer system 2200 and other devices attached to a network2285 (e.g., carrier or agent devices) or between nodes of computersystem 2200. Network 2285 may in various embodiments include one or morenetworks including but not limited to Local Area Networks (LANs) (e.g.,an Ethernet or corporate network), Wide Area Networks (WANs) (e.g., theInternet), wireless data networks, some other electronic data network,or some combination thereof. In various embodiments, network interface2240 may support communication via wired or wireless general datanetworks, such as any suitable type of Ethernet network, for example;via telecommunications/telephony networks such as analog voice networksor digital fiber communications networks; via storage area networks suchas Fibre Channel SANs, or via any other suitable type of network and/orprotocol.

Input/output devices 2250 may, in some embodiments, include one or moredisplay terminals, keyboards, keypads, touchpads, scanning devices,voice or optical recognition devices, or any other devices suitable forentering or accessing data by one or more computer systems 2200.Multiple input/output devices 2250 may be present in computer system2200 or may be distributed on various nodes of computer system 2200. Insome embodiments, similar input/output devices may be separate fromcomputer system 2200 and may interact with one or more nodes of computersystem 2200 through a wired or wireless connection, such as over networkinterface 2240.

As shown in FIG. 22 , memory 2220 may include program instructions 2222,which may be processor-executable to implement any element or actiondescribed above. In one embodiment, the program instructions mayimplement the methods described above. In other embodiments, differentelements and data may be included. Note that data may include any dataor information described above.

Those skilled in the art will appreciate that computer system 2200 ismerely illustrative and is not intended to limit the scope ofembodiments. In particular, the computer system and devices may includeany combination of hardware or software that can perform the indicatedfunctions, including computers, network devices, Internet appliances,PDAs, wireless phones, pagers, etc. Computer system 2200 may also beconnected to other devices that are not illustrated, or instead mayoperate as a stand-alone system. In addition, the functionality providedby the illustrated components may in some embodiments be combined infewer components or distributed in additional components. Similarly, insome embodiments, the functionality of some of the illustratedcomponents may not be provided and/or other additional functionality maybe available.

Those skilled in the art will also appreciate that, while various itemsare illustrated as being stored in memory or on storage while beingused, these items or portions of them may be transferred between memoryand other storage devices for purposes of memory management and dataintegrity. Alternatively, in other embodiments some or all of thesoftware components may execute in memory on another device andcommunicate with the illustrated computer system via inter-computercommunication. Some or all of the system components or data structuresmay also be stored (e.g., as instructions or structured data) on acomputer-accessible medium or a portable article to be read by anappropriate drive, various examples of which are described above. Insome embodiments, instructions stored on a computer-accessible mediumseparate from computer system 2200 may be transmitted to computer system2200 via transmission media or signals such as electrical,electromagnetic, or digital signals, conveyed via a communication mediumsuch as a network and/or a wireless link Various embodiments may furtherinclude receiving, sending or storing instructions and/or dataimplemented in accordance with the foregoing description upon acomputer-accessible medium. Generally speaking, a computer-accessiblemedium may include a non-transitory, computer-readable storage medium ormemory medium such as magnetic or optical media, e.g., disk orDVD/CD-ROM, volatile or non-volatile media such as RAM (e.g. SDRAM, DDR,RDRAM, SRAM, etc.), ROM, etc. In some embodiments, a computer-accessiblemedium may include transmission media or signals such as electrical,electromagnetic, or digital signals, conveyed via a communication mediumsuch as network and/or a wireless link

The methods described herein may be implemented in software, hardware,or a combination thereof, in different embodiments. In addition, theorder of the blocks of the methods may be changed, and various elementsmay be added, reordered, combined, omitted, modified, etc. Variousmodifications and changes may be made as would be obvious to a personskilled in the art having the benefit of this disclosure. The variousembodiments described herein are meant to be illustrative and notlimiting. Many variations, modifications, additions, and improvementsare possible. Accordingly, plural instances may be provided forcomponents described herein as a single instance. Boundaries betweenvarious components, operations and data stores are somewhat arbitrary,and particular operations are illustrated in the context of specificillustrative configurations. Other allocations of functionality areenvisioned and may fall within the scope of claims that follow. Finally,structures and functionality presented as discrete components in theexample configurations may be implemented as a combined structure orcomponent. These and other variations, modifications, additions, andimprovements may fall within the scope of embodiments as defined in theclaims that follow.

What is claimed is:
 1. A non-transitory, computer-readable, mediumstoring program instructions, that when executed on or across one ormore processors, cause the one or more processors to: receive compressedspatial information for a set of points in three-dimensional (3D) space;receive compressed attribute information for the set of points in 3Dspace, wherein the compressed attribute information comprises compressedattribute information for attributes of respective ones of the pointsincluded in a plurality of levels of detail; determine, using thecompressed spatial information, the plurality of levels of detail,wherein to determine the plurality of levels of detail the programinstructions cause the one or more processors to: reconstruct spatiallocations for the points of the set of points in 3D space bydecompressing the compressed spatial information; determine an orderedindex for the points of the set of points in 3D space, based onrespective spatial locations of the points of the set of points in 3Dspace; and sub-sample the index to determine respective ones of the setof points in 3D space to be included in respective ones of the pluralityof levels of detail; and determine, for respective ones of the pluralityof levels of detail, attribute values for respective points of the setof points in 3D space included in the respective level of detail basedon the compressed attribute information.
 2. The non-transitory,computer-readable, medium of claim 1, wherein to determine the orderedindex, the program instructions cause the one or more processors to:order the points in a Morton order determined using a space fillingcurve.
 3. The non-transitory, computer-readable, medium of claim 1,wherein to sub-sample the index to determine the respective ones of theset of points in 3D space to be included in respective ones of theplurality of levels of detail, the program instructions cause the one ormore processors to: sample the index according to a sampling rate,wherein every Nth point in the index is sampled for inclusion in a givenone of the plurality of levels of detail based on a given sampling rateto be used to select points to include in the given level of detail. 4.The non-transitory, computer-readable, medium of claim 1, wherein tosub-sample the index to determine the respective ones of the set ofpoints in 3D space to be included in the respective ones of theplurality of levels of detail, the program instructions cause the one ormore processors to: perform a distance-based sampling process using anearest neighbor search to determine the respective ones of the set ofpoints in 3D space to be included in the respective ones of theplurality of levels of detail.
 5. The non-transitory, computer-readable,medium of claim 4, wherein to perform the nearest neighbor search, theprogram instructions cause the one or more processors to: group sub-setsof the points of the set of points in 3D space into blocks; determine,for a given sub-set of points included in a given block, a distance froma point being evaluated to the block; and if the distance to the blockexceeds a threshold distance for inclusion in a given level of detail,refrain from calculating individual distances for individual ones of thepoints of the sub-set of points included in the block.
 6. Thenon-transitory, computer-readable medium of claim 1, wherein todetermine, for the respective ones of the plurality of levels of detail,the attribute values for the respective points of the set of points in3D space included in the respective level of detail, the programinstructions cause the one or more processors to: perform adistance-based prediction; and apply attribute correction valuesincluded in the receive compressed attribute information to attributevalues predicted based on the distance-based prediction.
 7. Thenon-transitory, computer-readable medium of claim 1, wherein todetermine, for the respective ones of the plurality of levels of detail,the attribute values for the respective points of the set of points in3D space included in the respective level of detail, the programinstructions cause the one or more processors to: apply a lifting schemeto predict attribute values for points included in two or more of theplurality of levels of detail; and apply attribute correction valuesincluded in the receive compressed attribute information to attributevalues predicted based on the lifting scheme.
 8. A non-transitory,computer-readable, medium storing program instructions, that whenexecuted on or across one or more processors, cause the one or moreprocessors to: receive information defining a set of points inthree-dimensional (3D) space, wherein the information includes, forrespective ones of the points, spatial information locating therespective point in 3D space and attribute information for therespective point; determine an ordered index for the points of the setof points in 3D space, based on respective spatial locations of thepoints of the set of points in 3D space; sub-sample the index todetermine respective ones of the set of points in 3D space to beincluded in respective ones of a plurality of levels of detail; anddetermine, for respective ones of the plurality of levels of detail,compressed attribute information for the respective ones of the pointsincluded in the respective ones of the plurality of levels of detail. 9.The non-transitory, computer-readable, medium of claim 8, wherein todetermine the ordered index, the program instructions cause the one ormore processors to: order the points in a Morton order determined usinga space filling curve.
 10. The non-transitory, computer-readable, mediumof claim 8, wherein to sub-sample the index to determine the respectiveones of the set of points in 3D space to be included in respective onesof the plurality of levels of detail, the program instructions cause theone or more processors to: sample the index according to a samplingrate, wherein every Nth point in the index is sampled for inclusion in agiven one of the plurality of levels of detail based on a given samplingrate to be used to select points to include in the given level ofdetail.
 11. The non-transitory, computer-readable, medium of claim 8,wherein to sub-sample the index to determine the respective ones of theset of points in 3D space to be included in respective ones of theplurality of levels of detail, the program instructions cause the one ormore processors to: perform a distance-based sampling process using anearest neighbor search to determine the respective ones of the set ofpoints in 3D space to be included in the respective ones of theplurality of levels of detail.
 12. The non-transitory,computer-readable, medium of claim 11, wherein to perform the nearestneighbor search, the program instructions cause the one or moreprocessors to: group sub-sets of the set of points in 3D space intoblocks; determine, for a given sub-set of points included in a givenblock, a distance from a point being evaluated to the block; and if thedistance to the block exceeds a threshold distance for inclusion in agiven level of detail, refrain from calculating individual distances forindividual ones of the points of the sub-set of points included in theblock.
 13. The non-transitory, computer-readable medium of claim 8,wherein to determine, for the respective ones of the plurality of levelsof detail, the compressed attribute information for the respectivepoints of the set of points in 3D space included in the respective levelof detail, the program instructions cause the one or more processors to:perform a distance-based prediction; and determine attribute correctionvalues to be included in the compressed attribute information based onthe distance-based prediction.
 14. The non-transitory, computer-readablemedium of claim 8, wherein to determine, for the respective ones of theplurality of levels of detail, the attribute values for the respectivepoints of the set of points in 3D space included in the respective levelof detail, the program instructions cause the one or more processors to:apply a lifting scheme to predict attribute values for points includedin two or more of the plurality of levels of detail; and determineattribute correction values to be included in the compressed attributeinformation based on attribute values predicted using the liftingscheme.
 15. A device, comprising: a memory storing program instructions;and one or more processors, wherein the program instructions, whenexecuted on or across the one or more processors, cause the one or moreprocessors to: receive compressed spatial information for a set ofpoints in three-dimensional (3D) space; receive compressed attributeinformation for the set of points in 3D space, wherein the compressedattribute information comprises compressed attribute information forattributes of respective ones of the points included in a plurality oflevels of detail; determine, using the compressed spatial information,the plurality of levels of detail, wherein to determine the plurality oflevels of detail the program instructions cause the one or moreprocessors to: determine an ordered index for the points of the set ofpoints in 3D space, based on respective spatial locations of the pointsof the set of points in 3D space as determined using the compressedspatial information; and sub-sample the index to determine respectiveones of the set of points in 3D space to be included in respective onesof the plurality of levels of detail; and determine, for respective onesof the plurality of levels of detail, attribute values for respectivepoints of the set of points in 3D space included in the respective levelof detail based on the compressed attribute information.
 16. The deviceof claim 15, wherein to determine the ordered index, the programinstructions cause the one or more processors to: order the points in aMorton order determined using a space filling curve.
 17. The device ofclaim 15, wherein to sub-sample the index to determine the respectiveones of the set of points in 3D space to be included in respective onesof the plurality of levels of detail, the program instructions cause theone or more processors to: sample the index according to a samplingrate, wherein every Nth point in the index is sampled for inclusion in agiven one of the plurality of levels of detail based on a given samplingrate to be used to select points to include in the given level ofdetail.
 18. The device of claim 15, wherein to sub-sample the index todetermine the respective ones of the set of points in 3D space to beincluded in respective ones of the plurality of levels of detail, theprogram instructions cause the one or more processors to: perform adistance-based sampling process using a nearest neighbor search todetermine the respective ones of the set of points in 3D space to beincluded in the respective ones of the plurality of levels of detail.19. The device of claim 15, wherein to determine, for the respectiveones of the plurality of levels of detail, the attribute values for therespective points of the set of points in 3D space included in therespective level of detail, the program instructions cause the one ormore processors to: perform a distance-based prediction; and applyattribute correction values included in the receive compressed attributeinformation to attribute values predicted based on the distance-basedprediction.
 20. The device of claim 15, wherein to determine, for therespective ones of the plurality of levels of detail, the attributevalues for the respective points of the set of points in 3D spaceincluded in the respective level of detail, the program instructionscause the one or more processors to: apply a lifting scheme to predictattribute values for points included in two or more of the plurality oflevels of detail; and apply attribute correction values included in thereceive compressed attribute information to attribute values predictedbased on the lifting scheme.